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Abstract: 


Course  management  software  enables  faculty  members  to  learn  one  software 
package  for  web-based  curriculum,  assessment,  synchronous  and  asynchronous 
discussions,  collaborative  work,  multimedia  and  interactive  resource  development.  There 
are  as  many  as  109  different  course  management  software  packages  on  the  market  and 
several  studies  have  evaluated  and  compared  various  aspects  of  these  tools.  However, 
these  studies  generally  focused  on  checklists  of  what  these  products  can  do,  not  what 
these  products  need  to  do,  or  can  do  well. 

In  addition,  there  is  little  data  to  suggest  that  the  pages  created  by  these  software 
packages  were  easy  to  use  by  the  students,  or  that  the  use  of  these  tools  by  faculty 
increased  the  quality  of  instruction  experienced  by  the  students.  Faculty  and  students 
have  very  different  requirements  for  this  one  product,  faculty  need  to  be  able  to  develop 
resources  quickly  and  efficiently  using  this  software,  while  students  need  to  be  able  to 
find  and  access  the  course  materials  their  instructor  has  assigned.  Therefore,  these 
analyses  may  be  of  limited  predictive  power  in  determining  the  applicability  of  different 
course  management  software  for  deployment  by  educational  institutions. 

The  objective  of  this  study  was  to  evaluate  the  usability  and  usefulness  of  course 
management  software  to  support  traditional  classroom  instruction  from  both  the  faculty 
and  student  perspective.  This  study  was  done  in  two  parts,  the  first  part  asked  participants 
with  no  experience  using  course  management  software  to  evaluate  several  packages  and 
choose  the  one  they  preferred,  and  the  second  part  was  a  follow-up  analysis  after  both 
faculty  and  students  had  used  the  software  for  an  entire  semester. 

Faculty  members  and  cadets  found  that  course  management  software  (CMS) 
provided  a  satisfactory  mechanism  for  web-enabled  curriculum  delivery  to  supplement 
traditional  classroom  instruction  at  USAFA.  If  we  had  only  focused  on  checklists  of  what 
the  software  products  could  do,  then  Blackboard  Courseinfo  and  WebCT  would  have 
been  considered  equal,  and  Intralearn  would  have  been  a  distant  third.  However, 
Blackboard  Courseinfo  received  higher  scores  than  the  other  products  when  ratings  were 
based  on  user  analysis  of  ease  of  use  and  usefulness. 

Many  of  the  features  found  in  these  products  such  as  discussion  groups,  student 
web  pages  and  collaborative  work  areas  that  contributed  to  their  high  number  of  features 
were  not  widely  used,  nor  deemed  important  by  both  faculty  and  cadets.  Although  some 
faculty  may  use  these  features  in  the  future  (and  subsequently  cadets  will  use  them)  as 
they  become  more  comfortable  and  familiar  with  the  software  and  pedagogy,  at  this  time 
a  CMS  with  an  easy  to  use  interface  that  contains  a  grade  book,  automated  quizzes  and  a 
place  to  put  announcements  and  course  documents  should  be  preferred  to  one  that 
contains  many  collaboration  features  yet  also  has  a  difficult  navigational  interface  or  hard 
to  use  development  tools. 


Many  faculty  members  chose  to  use  Blackboard  Courseinfo  for  a  follow  on 
semester  for  a  more  in-depth  analysis.  Although  faculty  found  Courseinfo  served  most  of 


their  needs,  many  indieated  that  it  lacked  some  key  features  necessary  for  deployment  at 
USAFA,  for  example  there  was  no  way  to  divide  grade  books,  announcements  or 
documents  into  multiple  sections  of  the  same  course  without  duplicating  the  entire 
course.  Similarly,  it  was  not  possible  to  share  calendars,  quiz  questions,  handouts,  or 
content  among  instructors  teaching  the  same  course.  Therefore,  this  software  seems  best 
suited  for  providing  web-enabled  instructional  support  to  a  faculty  member  who  is 
teaching  one  section  of  a  single  course,  and  who  does  not  share  large  amounts  of  content 
with  instructors  teaching  other  sections. 

As  predicted,  the  faculty  and  cadets  had  different  requirements  of  the  CMS 
product  for  developing  and  accessing  resources,  respectively.  Therefore,  if  only  the 
faculty’s  requirements  are  considered,  or  if  the  product  is  not  evaluated  by  all  of  the  user 
groups,  the  probability  of  purchasing  a  sub-optimal  product  for  a  majority  of  the  users 
increases.  Although  the  requirements  as  stated  by  the  faculty  and  cadets  who  had  been 
using  a  CMS  for  an  entire  semester  were  more  similar  than  those  gathered  from  the  naive 
users,  often  enterprise  CMS  purchase  decisions  are  made  prior  to  the  purchasing  of  any 
system. 


The  results  obtained  in  this  part  of  the  study  will  help  us  determine  the 
requirements  for  procuring  a  course  management  software  system  for  use  here  at 
USAFA.  Giving  faculty  an  efficient  means  of  entering  their  homework  assignments  into 
a  database  system  is  the  first  and  probably  most  important  step  to  achieving  the  vision  of 
an  integrated  portal  system  that  includes  the  resources  the  cadets  need  most,  access  to 
homework  and  reading  assignments,  course  materials  and  the  ability  to  monitor  their 
progress  through  checking  their  grades. 


Keywords:  web  design  usability,  course  management  software 


Section  1.  Introduction 


In  the  past  few  years,  the  World  Wide  Web  (WWW  or  web)  and  web 
teehnologies  have  beeome  aeeepted  and  valuable  tools  for  delivering  eontent  in  distance 
learning,  as  well  as  a  means  to  supplement  traditional  classroom  instruction  (Brown, 
2000;  Halloran,  1999a;  Supinski,  1999).  The  use  of  the  web  to  support  traditional 
classroom  instruction  ranges  from  a  delivery  mechanism  for  readings  and  course  content 
to  a  means  of  expanding  the  curriculum  outside  the  classroom  walls,  as  well  as  a  means 
for  engaging  students  in  interactive  learning  environments  (Novak  et  al,  1999). 

However,  one  of  the  drawbacks  to  web-enabling  courses  is  that  it  may  require 
faculty  and  students  to  acquire  new  technical  skills,  such  as  the  ability  to  write  HTML 
and  other  web  programming  language  codes  or  proficiency  with  HTML  editor  software 
programs.  In  these  cases,  the  technology  can  overshadow  the  teaching  and  learning  of  the 
course  content  (Poe,  2000;  Taylor,  2000),  and  converting  a  course  to  a  web-enabled 
platform  can  be  very  labor  intensive  (Campbell,  2000). 

One  possible  way  to  circumvent  these  drawbacks  is  to  use  course  management 
system  or  learning  management  system  packages,  which  use  templates  specifically 
designed  for  web-enabled  instruction.  Course  management  systems  (CMS)  incorporate 
many  different  development  tools  into  a  seamless  interface  for  web  based  course  content. 
By  uploading  course  handouts,  images  and  fdes  already  created  using  other  software 
packages  such  as  Microsoft  (MS)  Word  or  Excel  into  the  system,  a  faculty  member  can 
develop  a  web  presence  without  the  knowledge  of  HTML  or  other  languages. 

Course  management  software  enables  faculty  members  to  learn  one  software 
package  for  web-based  curriculum,  assessment,  synchronous  and  asynchronous 
discussions,  collaborative  work,  multimedia  and  interactive  resource  development. 
Tracking  features  enable  instructors  to  examine  the  use  of  these  online  resources,  which 
can  be  used  to  make  inferences  about  the  interest  and  difficulty  of  the  content  as  well  as 
an  individual  students’  level  of  effort.  Although  some  faculty  members  may  prefer  to  use 
different  software  for  different  applications,  incorporating  all  tools  into  one  package  may 
decrease  the  need  to  train  faculty  on  a  variety  of  software  products  (Hazari,  2001). 

Deploying  a  CMS  at  the  institutional  level  offers  students  a  consistent,  virtual 
interactive  environment  which  promotes  learning  and  decreases  the  number  of  systems  a 
student  has  to  learn  to  accomplish  routing  tasks,  such  as  finding  their  homework 
assignments  or  an  instructor’s  office  hours  (Hazari,  2001).  Another  advantage  is  the 
underlying  database  structure  of  the  system  (e.g.  Oracle,  MySql,  etc).  A  homework 
assignment  entered  by  a  faculty  member  into  their  course  can  be  used  to  populate  a 
student  home  page  through  a  learning  management  portal  system  (Halloran,  1999b;  and 
in  prep).  At  the  same  time,  it  is  important  that  the  CMS  does  not  drive  the  pedagogy  and 
limit  the  creativity  of  an  instructor  developing  sophisticated  course  content  using  a 
variety  of  different  mediums. 


There  are  as  many  as  109  different  course  management  software  packages  on  the 
market  (Landon,  2000),  and  several  studies  have  evaluated  and  compared  various  aspects 
of  these  tools  (Gray,  1998;  Hazari,  2001;  InfoWorld,  1998;  Kristapiazzi,  1998;  Marshall, 
1998).  However,  these  studies  generally  focused  on  checklists  of  what  these  products  can 
do,  not  what  these  products  need  to  do,  or  can  do  well.  For  example,  packages  were 
evaluated  by  whether  or  not  they  had  an  asynchronous  discussion  feature,  not  whether  the 
discussion  group  feature  was  easy  to  use  or  useful  to  faculty  members  in  teaching  their 
courses.  Focusing  on  what  a  product  can  do  is  also  of  limited  value  given  the  rapid 
development  and  changes  in  product  features  in  this  market. 

In  addition,  there  is  little  data  to  suggest  that  the  pages  created  by  these  software 
packages  were  easy  to  use  by  the  students,  or  that  the  use  of  these  tools  by  faculty 
increased  the  quality  of  instruction  experienced  by  the  students.  Faculty  and  students 
have  very  different  requirements  for  this  one  product,  faculty  need  to  be  able  to  develop 
resources  quickly  and  efficiently  using  this  software,  while  students  need  to  be  able  to 
find  and  access  the  course  materials  their  instructor  has  assigned.  Therefore,  these 
analyses  may  be  of  limited  predictive  power  in  determining  the  applicability  of  different 
course  management  software  for  deployment  by  educational  institutions. 

1.1  Objectives  of  the  study 

The  objective  of  this  study  was  to  evaluate  the  usability  and  usefulness  of  course 
management  software  to  support  traditional  classroom  education  from  both  the  faculty 
and  student  perspective.  Faculty  were  asked  to  rate  the  different  software  packages  not 
only  on  how  easy  they  were  to  use,  but  how  useful  the  features  offered  were  to  classroom 
instruction.  Students  were  asked  to  evaluate  the  resulting  web-enabled  courses  as  to  their 
ease  of  navigation  and  finding  resources  as  well  as  how  useful  they  found  the  different 
features  packaged  within  the  products.  This  study  was  done  in  two  phases,  the  first  phase 
asked  participants  with  no  experience  using  course  management  software  to  evaluate 
several  packages  and  choose  the  one  they  preferred,  and  the  second  phase  was  a  follow¬ 
up  analysis  after  both  faculty  and  students  had  used  the  software  for  an  entire  semester. 

Section  2.  Course  Management  Software  Usability  Tests 

2.1  Background  information 

The  United  States  Air  Force  Academy  (USAFA)  is  a  resident  institution  of  higher 
education.  Cadets  are  required  to  live  in  dormitories  and  attend  class,  and  faculty 
members  are  regularly  available  to  meet  with  cadets  during  extended  office  hours.  There 
are  19  academic  departments  offering  31  different  majors  and  over  500  different  courses. 
Although  USAFA  does  not  have  a  distance-learning  program,  nor  offer  distance-learning 
courses,  the  instructors  are  encouraged  to  use  distance  learning  and  other  information 
technologies  to  supplement  the  use  of  the  traditional  classroom  education,  when 
appropriate.  The  Academy  encourages  faculty  to  make  extensive  use  of  active-learning 
strategies  for  the  classroom. 


The  Academy  first  began  issuing  computers  to  cadets  in  1985,  and  since  1989  all 
cadets  have  had  computers  in  their  dormitory  rooms.  E-mail  has  been  in  place  at  USAFA 
since  1993,  and  instructors  have  been  able  to  place  class  handouts  on  a  network  of  shared 
computer  drives  accessible  by  all  cadets  since  1994.  The  Academy  was  linked  to  the 
internet  in  1996,  and  many  faculty  now  use  the  World  Wide  Web  in  their  classrooms  as  a 
teaching  tool. 

With  the  ubiquitous  computing  environment  here  at  USAFA,  there  is  ample 
opportunity  for  faculty  members  to  develop  resources  for  their  courses.  And  without 
coordination  among  departments  and  instructors  on  the  development  of  web-enabled 
courses,  cadets  could  end  up  having  to  learn  as  many  different  systems  as  they  take 
courses.  There  is  also  a  minimum  of  24%  turnover  of  the  faculty  and  cadets  every 
academic  year.  Any  system  deployed  here  must  be  intuitive  in  nature,  and  require 
minimum  training  for  both  sets  of  users,  unless  provisions  are  made  to  accommodate 
extensive  training  time  and  costs. 

While  it  would  be  possible  to  author  a  custom  solution  CMS,  the  development 
costs  of  an  enterprise  level  database  driven  solution  are  many  times  more  than  what  the 
commercial  off  the  shelf  (COTS)  products  charge  for  license  agreements  over  many 
years.  In  addition,  many  textbook  publishing  companies  are  now  forming  alliances  with 
companies  that  produce  CMS  software  and  making  these  supplemental  resources 
available  to  faculty  and  students  who  buy  their  textbooks.  In  order  to  take  advantage  of 
these  services,  we  felt  that  an  exploration  of  COTS  CMS  solutions  was  a  logical  first 
step. 


Therefore,  to  accommodate  the  diversity  of  courses  taught  here  at  the  Academy, 
we  are  interested  in  a  system  that  allows  for  flexibility  as  to  the  type  of  content  that  can 
be  uploaded  by  the  faculty.  However,  at  the  same  time,  this  system  must  be  an  easy 
developmental  tool  for  the  faculty,  as  well  as  provide  a  pedagogically  sound  and  intuitive 
navigational  structure  for  the  cadets.  Therefore,  the  objective  of  this  portion  of  the  study 
was  to  evaluate  different  commercial  off  the  shelf  CMS  products  to  begin  to  understand 
the  essential  components  and  requirements  necessary  for  deployment  of  an  enterprise 
level  course  management  system  at  USAFA. 

2.2  Methods 

2.2.1  Selection  of  software 

For  this  study,  6  commercial  software  packages  were  scored  using  dichotomous 
(yes/no)  variables  as  to  whether  they  have  certain  features  or  allow  for  the  development 
of  key  course  components.  Although  some  of  the  features  evaluated  during  this  process 
had  to  do  with  the  administration  of  the  software,  1  focused  on  those  features  that  were 
user  interface  components.  The  two  packages  with  the  most  features  were  further 
evaluated  for  usability.  The  faculty  evaluated  the  product  as  a  development  tool,  whereas 
the  students  evaluated  the  resulting  courses  put  together  by  the  development  tool. 


2.2.2  Participants 

Faculty  and  cadet  members  from  the  United  States  Air  Foree  Aeademy  (USAFA) 
volunteered  to  partieipate  in  the  study  in  response  to  an  email  query  sent  to  all  aeademie 
departments.  Prior  to  beginning  the  study  they  were  given  a  survey  to  assess  their 
eomputer  expertise  including  any  prior  experience  building  web  pages  and  using  the 
internet,  both  at  home  and  as  part  of  their  eourse  preparation.  They  were  also  asked  to 
rate  the  features  that  were  used  in  the  diehotomous  rankings  of  eourse  management 
software  as  to  their  usefulness  in  teaehing  and  learning,  prior  to  being  exposed  to  the 
eourse  management  shell  software. 

2.2.2  Instructor  choice  experiments 

Instruetor  study  partieipants  were  asked  to  develop  a  web  site  with  a  syllabus, 
eourse  information,  homework  assignments  and  quizzes  using  two  out  of  three  CMS 
being  evaluated.  Both  the  software  paekages  and  the  order  in  whieh  they  were  presented 
to  the  instruetors  was  randomized.  Instruetors  ereated  eourse  eontent  using  the  web 
authoring  feature  of  the  software  package  itself,  and  imported  pre-manufactured  web 
pages  built  in  plain  text  HTML,  MSFrontPage  and  Maeromedia’s  Dreamweaver.  They 
were  also  given  MSWord  doeuments  and  MSExeel  spreadsheet  fdes  to  upload  to 
determine  the  eross  eompatibility  of  this  system  with  other  software  produets  already  in 
use  at  USAFA.  The  eourse  eontent,  handouts,  syllabus,  quiz  questions  and  quiz  answers 
were  identieal  and  supplied  to  the  instructor.  Therefore  the  ease  or  diffieulty  of  putting 
together  the  eourse  web  site  would  be  software  dependent,  not  dependent  on  the 
ereativity  of  the  instruetor. 

After  they  had  evaluated  two  CMS  paekages,  faeulty  members  were  asked  to  do  a 
side-by-side  eomparison  of  eaeh  produet  for  18  key  eomponents.  Data  eolleeted  from  the 
instructors  included  time  on  task,  ease  of  use,  and  the  usefulness  of  the  tasks  using  a  1- 
100  visual  analog  seale  (Miller  1999).  They  were  then  asked  to  select  one  paekage  as  the 
best  overall  and  to  use  that  CMS  during  the  Fall  ’00  semester. 

2.2.4  Cadet  choice  experiments 

Cadets  eaeh  evaluated  three  eourses  that  were  built  by  the  same  person,  using  the 
same  software  paekages  evaluated  by  the  faeulty.  These  eourses  were  identieal  with 
respeet  to  eourse  eontent,  handouts,  quiz  questions  and  answers,  assignments  and  other 
information.  Eaeh  eadet  evaluated  two  of  the  three  eourses.  The  eourses  they  evaluated  as 
well  as  the  order  in  which  they  were  presented  was  randomized. 

Cadets  were  asked  to  navigate  to  specifie  areas  within  the  eourse,  take  quizzes, 
participate  in  discussions,  collaborate  on  group  assignments,  find  their  grade,  and 
perform  other  tasks  neeessary  to  test  the  functional  capability  of  the  CMS.  They  were 
also  asked  to  make  predietions  on  where  they  would  find  key  eourse  doeuments.  After 
they  evaluated  two  eourses,  eadets  were  asked  to  eomplete  a  side-by-side  eomparison  of 
the  courses  of  eaeh  product  for  12  key  tasks.  Data  eolleeted  included  time  on  task,  ease  of 
use  and  usefulness  of  the  task  using  a  1-100  visual  analog  scale  (Miller,  1999).  Cadets 
also  seleeted  one  eourse  as  their  preferred  eourse,  but  due  to  practieal  eonsiderations  did 
not  use  it  for  an  entire  semester. 


2.2.5  Data  collection  and  analysis 

Data  for  both  the  faculty  and  cadets  were  collected  by  self-report  on  surveys  and 
entered  into  a  MS  Excel  spreadsheet.  Summary  statistics  were  calculated  using  MSExcel 
and  are  reported  as  mean  +  sd  unless  otherwise  noted.  Data  were  imported  into  SPSS  for 
further  analysis.  Non-parametric  and  parametric  within  subjects  comparisons  were 
performed  to  compare  scores  assigned  to  software  packages  by  faculty  and  cadets. 
Unpaired  t-tests  were  performed  to  examine  differences  between  computer  expertise  of 
faculty  that  finished  the  study  with  faculty  that  did  not  finish  the  study,  and  for 
comparisons  between  faculty  and  cadets. 

WebCT  and  Blackboard’s  Courseinfo  were  housed  on  a  Sun  Ultra  Sparc  5, 
Intralearn  software  was  housed  on  a  400  MHz  NT  4.0  server  with  256  MB  RAM,  and  a 
4x9  GB  SCSI  RAID  (RAID  5)  located  in  the  Institute  for  Information  Technology 
Applications  laboratory  in  Eairchild  Hall  at  USAEA.  The  web  pages  were  accessed 
through  the  academy’s  intranet  bypassing  the  proxies  for  efficiency  and  speed. 

2,3  Results 

2.2. 1  Selection  of  software  for  further  evaluation 

Six  software  packages  were  dichotomously  scored  in  May  2000  as  to  whether 
they  contained  key  pedagogical  and  course  management  system  components  (Table  1). 
The  software  packages  were  WebCT,  Blackboard  Courseinfo  Level  1,  TopClass,  Mad 
Duck,  Intralearn,  and  Virtual  University.  Scoring  was  done  on  examination  copies  of  the 
software  either  hosted  locally  or  on  the  company’s  web  servers.  These  scores  were 
compared  to  other  published  dichotomous  scorings  of  a  similar  nature  (Gray,  1998; 
Hazarri,  1999;  Kristapiazzi,  1999;  Marshall,  1998;  Landon,  2000)  for  verification.  The 
products  with  the  two  highest  scores  (WebCT  and  Courseinfo)  were  chosen  for  further 
evaluations  because  they  contained  the  widest  variety  of  tools  for  the  broadest  evaluation. 
Intralearn  was  also  chosen  because  its  product  uses  Microsoft  Office  products  for  many 
of  its  development  tools.  MS  Office  is  the  standard  suite  of  tools  on  the  USAEA 
computer  desktop  and  therefore  would  provide  a  control  for  familiarity. 


BB 

ICT 

Itc 

MD 

■>- 

VU 

Table  1:  Dichotomous  scoring  of  web  course  management  software  Cl  =  Courseinfo, 
CT  =  WebCT,  TC  =  TopClass,  MD  =  Mad  Duck,  IL  =  Intralearn,  VU  =  Virtual 
University. 


General  Features 


Static  toolbar 


Table  of  Contents 


Customized  "look  and  feel"  (colors,  icons,  logos,  etc) 


Automated  glossary  tool 


Automated  link  to  course  material  content 


Automated  indexing  tool 


Search  tool  for  course  material 


Student  can  make  private  annotations  of  course  material 


Integrated  Calendar  tool 


Multi  language  support 


Instructor  can  define  groups  of  students 


-  Can  assign  specific  material  to  individual  or  group  of  students  |x 


-  Collaborative  work  area  for  group 


-  Group  presentation  area 


Individual  presentation  area/homepage 


Student  file  upload  capability/instructor  comments 


Conforms  to  IMS  specifications 


Conforms  to  W3C  specifications 


Authoring  features 


Does  not  require  knowledge  of  HTML 


Allows  Java  applets 


Requires  plug-ins 


Has  drag  and  drop  authoring  features 


Allows  Macromedia  files 


Allows  Frontpage  files 


Allows  links  to  custom  developed  pages 


Allows  custom  graphics 


Course  Management  Features 


Class  list  can  be  entered  one  student  at  a  time 


Class  list  can  be  uploaded  as  a  file 


Class  lists  can  be  presented,  saved,  and  printed  using  Excel 


Online  student  manual 


Online  instructor  manual 


Course  can  be  downloaded  for  safekeeping 


Courses  can  easily  be  moved  from  one  server  to  another 


File  upload  capability  from  desktop 


Student  tracking  features 


-  can  track  how  often  student  accesses  pages 


-  can  track  when  student  accesses  pages 


-  can  track  time  spent  on  each  page 


no  no  no  no 


X 


X 


X 


no  ho 


Communication  Features 


One  to  one  course  email 


One  to  many  course  email 


Searchable  asynchronous  discussion 


Logged  synchronous  discussion 


Vitual  field  trips  within  discussion  pages 


Shared  whiteboard 


Adaptable  for  desktop  videoteleconferencing 


BB  CT  TC  MD  ML  VU 


Grading  Features 


Student  access  to  progress  data  available 


Ability  to  add  offline  grades 


Grade  statistics  and/or  histograms 


Instructor  comments  available  with  grade 


Scores  can  be  emailed  to  instructor 


Scores  can  be  stored  on  server 


Scores  can  be  exported  into  Excel 


BB  CT  TC  MD  ML 


Quizzing  features 


Quizzes  automatically  graded  and  entered  into  gradebook 


Allows  for  the  following  automatically  graded  question  types: 


-  Tme  -False/Multiple  choice 


-  Fill  in  the  blank 


-  List  matching 


-  Essay  questions 


-  Imagemap  (click  on  correct  part  of  image) 


-  Short  answer 


Questions  can  have  multiple  correct  answers 


Can  use  a  mixture  of  question  types  on  a  single  quiz 


One  question  at-a-time  testing  capability 


Question  file  upload  capability 


Customized  feedback 


Redirect  path  dependent  on  question  answers 


Timed  quizzes 


Delivered  on-line  on  a  predetermined  time  and  day 


Supports  graphics  files  adjacent  to  quiz  question 


Supports  both  tutorial  and  real  exam  scenarios 


Random  assignment  of  questions  to  exams 


Allows  weighting  of  questions  so  students  get  equal  quizzes 


Has  database  application  for  quiz  question  bank 


BB  CT  TC  MD  ML 


Unix 


NT 


Macintosh 


Solaris 


Linux 


Other 


TC 

MD 

X 

X 

X 

X 

X 

X 

Total 

60 

59 

42 

43 

36 

17 

2.3.2  Participants 

Twenty-eight  faeulty  members  from  the  United  States  Air  Foree  Academy 
volunteered  to  evaluate  the  course  management  shell  software  during  July  and  August  of 
2000.  These  faculty  were  a  representative  sample  of  all  USAFA  faculty  and  consisted  of 
members  from  all  four  academic  divisions  and  staff  (Table  2).  There  were  six  females 
and  22  males  whose  average  age  was  38.4  +  8.9.  Fifteen  of  these  participants  had 
Master’s  degrees  (MA/MS/MPH)  and  12  had  a  Ph.D  or  other  terminal  degree. 


Particioants 

USAFA  Faculty 

N 

% 

N 

% 

Male 

22 

78.6 

429 

81.0 

Female 

6 

21.4 

102 

19.0 

Military 

17 

60.7 

398 

75.0 

Civilian 

11 

39.3 

133 

25.0 

Academic  Divisions: 

Humanities 

6 

21.4 

104 

19.6 

Social  Sciences 

6 

21.4 

172 

32.3 

Basic  Sciences 

7 

25.0 

135 

25.5 

Enqineerinq 

6 

21.4 

120 

22.6 

Other 

3 

10.8 

Total 

28 

100.0 

531 

100.0 

Table  2:  Demographics  of  the  faculty  participants  compared  to  the  demographic  makeup 
of  all  USAFA  faculty. 


Thirty  cadets  from  USAFA  volunteered  to  evaluate  courses  produced  by  the 
course  management  shell  software  during  October  2000.  These  cadets  were  a 
representative  sample  of  all  cadets  at  USAFA  with  three  of  the  four  academic  divisions 
and  all  class  years  present  (Table  3).  The  average  age  of  all  cadets  was  19.3  +  1.2  and 
there  were  four  females  and  22  males  in  the  group. 


Table  3:  Demographic  makeup  of  cadet  participants  compared  to  the  USAFA  cadet 
student  body.  First  class  cadets  are  equivalent  to  seniors  at  civilian  universities,  second 
class  to  juniors,  third  class  to  sophomores  and  fourth  class  to  freshmen  students. 


Particioants 

All  Cadets 

N 

% 

N 

% 

Male 

26 

89.7 

3557 

84.7 

Female 

3 

10.3 

643 

15.3 

Year  at  the  Academy 

First  Class 

4 

13.8 

874 

20.8 

Second  Class 

7 

24.1 

962 

22.9 

Third  Class 

10 

34.5 

1147 

27.3 

Fourth  Class 

9 

31.0 

1218 

29.0 

Academic  Division: 

Humanities 

0 

0.0 

239 

5.7 

Social  Sciences 

10 

34.5 

1189 

28.3 

Basic  Sciences 

8 

27.6 

521 

12.4 

Enqineerinq 

6 

20.7 

802 

19.1 

Undecided 

6 

20.7 

1142 

27.2 

Interdisciplinary 

307 

7.3 

Total 

29 

100.0 

4200 

100.0 

2.3.3  Computer  experience 

Faculty  participants  were  experienced  computer  users  familiar  with  web 
browsing.  The  average  participant  had  12.7  +  4.8  years  experience  using  computers,  with 

5.6  +  7.3  years  experience  programming  computers.  They  averaged  5.1  +  1.8  years  of 
web  browsing  experience  and  routinely  browse  the  web  5.1  +  4.4  hours  a  week  for  both 
professional  and  personal  use  (Table  4). 

Faculty  participants  were  familiar  with  web-based  curriculum  materials.  Seventy- 
eight  percent  of  the  initial  participants  reported  having  used  web-based  curriculum 
support  materials  at  some  time  in  the  past,  with  59%  of  the  participants  using  web-based 
materials  for  the  Fall  ‘00  semester.  Of  those  who  used  web-based  materials,  over  half 
reported  using  resources  they  developed  (31%  of  all  participants).  Twenty-two  percent 
have  maintained  a  personal  web  server  and  30%  of  all  participants  maintain  a  personal 
website  with  an  internet  service  provider  outside  USAFA. 

Of  those  who  have  developed  their  own  web-based  content,  they  reported  1.5  + 

1.7  years  experience  developing  those  materials.  However,  the  faculty  had  little 
experience  using  HTML  authoring  tools  (0.6  +  0.9  years  experience)  such  as  MS 


Frontpage,  Netscape  Composer  and  Macromedia  Dreamweaver^'-^^.  Faculty  members 
reported  no  experience  using  interactive  development  tools  such  as  Allaire’s  Cold  Fusion, 
Microsoft  Visual  Studio,  Macromedia  Director  or  Macromedia  Attain  objects.  The 
faculty  also  had  no  experience  using  any  CMS  products  such  as  those  being  tested  in  this 
study  (Table  4). 

Table  4:  A  comparison  of  USAFA  faculty  (n  =  28)  and  cadet  (n  =  30)  computer 
experience. 


Task 

Faculty 

Cadets 

t 

df 

P 

Years  usina  comouters 

12.7 

9.7 

2.73 

56 

0.008 

Years  web  browsina 

5.1 

5.5 

0.89 

56 

0.375 

Flours  oer  week  browsina  the  web 

5.1 

11.7 

3.66 

0.001 

Years  develooina  web  content 

1.5 

1.5 

0.01 

56 

0.993 

Years  comouter  oroarammina 

5.6 

2.0 

2.50 

0.018 

Years  teachina  with  www-based  curriculum  materials^ 

5.1 

Years  develooina  www-based  curriculum  materials^ 

1.8 

Number  of  courses  usina  www-based  curriculum  materials^ 

3.0 

Maintain  a  oersonal  web  server  at  USAFA  (oercentl 

18.0 

10.0 

0.86 

56 

0.395 

Maintain  a  oesonal  web  site  outside  USAFA  (oercentl 

30.0 

40.0 

0.81 

55 

0.422 

Years  oroarammina  FTTML 

0.3 

0.9 

1.63 

0.111 

Years  oroarammina  JAVA 

0.1 

0.3 

1.13 

0.269 

Years  oroarammina  iavascriot 

0.0 

0.3 

1.80 

0.081 

Years  usina  FTTML  develooment  software^ 

0.4 

0.5 

n.s 

Years  usina  web  develooment  software^ 

0.0 

0.1 

n.s 

Web-based  Collaboration  Tools^ 

0.1 

0.0 

n.s 

Course  Manaqement  Tools'^ 

0.0 

0.0 

n.s 

'statistical  tests  were  not  performed  due  to  a  lack  of  comparable  question 
^statistical  tests  were  not  performed  due  to  inadequate  sample  size 
^the  degrees  of  freedom  were  adjusted  for  unequal  variance  among  groups 

Cadet  participants  were  very  familiar  with  using  web-based  curriculum  materials. 
They  reported  that  3.0  +  1.3  of  their  instructors  used  web-based  curriculum  support 
materials  for  their  courses  in  Fall  ’00.  Cadets  have  1.5  +  2.0  years  experience  developing 
their  own  web-based  content.  Twenty  percent  have  at  one  time  maintained  a  personal 
web  server  and  40%  of  all  the  cadets  reported  that  they  maintain  a  personal  website  with 
an  internet  service  provider  outside  USAFA  (Table  4). 

Cadets  had  little  experience  using  HTML  authoring  tools  (0.3  ±1.1  years)  such  as 
MS  Frontpage,  Netscape  Composer  and  Macromedia  Dreamweaver,  and  no  experience 
using  interactive  development  tools  such  as  Allaire’s  Cold  Fusion,  Microsoft  Visual 
Studio,  Macromedia  Director  or  Macromedia  Attain  objects.  The  cadets  had  no 


Microsoft  Frontpage  has  been  included  as  part  of  the  MS  Office  product  suite  available  to  all  Faculty 
and  Cadets  since  Fall  1999. 


experience  with  web-based  curriculum  resources  built  by  CMS  products  such  as  those 
being  tested  for  this  study. 


When  faculty  members’  computing  experience  was  compared  to  cadet  computing 
experience,  faculty  members  had  significantly  more  experience  using  and  programming 
computers  than  did  cadets.  However,  cadets  spend  significantly  more  time  browsing  the 
internet  each  week  than  do  faculty.  There  was  no  significant  difference  between  cadets 
and  faculty  in  the  number  of  years  each  group  reported  browsing  the  web,  or  developing 
WWW  content. 

2.3.4  Faculty  who  finished  compared  to  those  who  did  not  finish  the  experiments. 

Of  the  28  faculty  members  who  started  the  experiments,  15  faculty  finished  the 
experiments  and  13  of  the  faculty  did  not  finish  the  experiments.  Faculty  who  finished 
the  experiments  did  not  have  significantly  more  computer  or  web  development 
experience  than  those  faculty  who  did  not  finish  the  experiments  for  1 1  different 
experience  parameters  evaluated  (Table  5).  Faculty  who  did  not  finish  the  experiments 
cited  personal  time  management  issues  as  the  main  reason  for  discontinuing  the 
experiments. 

Table  5:  Comparison  of  computer  experience  for  faculty  who  finished  the  evaluation  of 
the  learning  management  software  (n=15),  with  faculty  who  did  not  finish  (DNF)  the 
evaluation  (n=13). 


Variable 

Finished 

DNF 

t 

df 

P 

Years  usinq  computers 

12.4 

13.0 

0.32 

26 

0.750 

Years  web  browsinq 

5.5 

4.7 

1.24 

26 

0.224 

Years  developinq  web  content 

1.4 

1.5 

0.15 

26 

0.886 

Years  usinq  ftp 

2.1 

1.9 

0.19 

26 

0.852 

Years  computer  proqramminq 

5.9 

5.2 

0.23 

26 

0.823 

Years  developinq  www-based  curriculum 

0.9 

0.7 

0.68 

26 

0.299 

Maintain  a  personal  web  server  at  USAFA 

13.0 

23.0 

0.65 

26 

0.520 

Maintain  a  pesonal  web  site  outside  USAFA  (%) 

33.0 

23.0 

0.58 

26 

0.566 

Years  proqramminq  HTML 

0.4 

0.5 

0.32 

26 

0.755 

Years  proqramminq  JAVA 

0.0 

0.1 

0.55 

26 

0.590 

Years  proqramminq  iavascript 

0.1 

0.3 

0.82 

26 

0.421 

2.3.5  Rankings  of  selection  criteria  by  faculty  and  cadets 

When  faculty  and  cadets  were  asked  to  rank  the  features  used  to  evaluate  the 
course  management  software  packages  (Table  1),  they  gave  most  of  the  features  similar 
scores.  The  Grand  mean  rating  given  to  all  the  features  by  faculty  was  4.6  +  0.70  and  for 
cadets  was  4.1  +  0.82  on  a  6  point  scale,  where  1  indicated  that  the  feature  was  of  very 
little  importance,  and  6  was  very  important  or  a  “must  have”.  Cadets  and  faculty 
disagreed  on  which  features  were  more  and  less  important,  with  the  exception  of  one 
feature  in  each  category  (Table  6  and  7). 


Nine  features  were  rated  5.3  or  higher  and  considered  to  be  more  important  to 
faculty  developing  curriculum.  Most  of  these  features  were  course  administration  features 
or  the  ability  to  include  HTML  content  developed  independently  of  CMS  software  (Table 
6).  Five  features  received  a  score  of  3.9  or  below  and  were  considered  less  important 
including  the  ability  to  conform  to  external  web  development  specifications  such  as  IMS, 
the  use  of  discussion  groups,  video-  teleconferencing  and  virtual  field  trips.  The  ability  to 
support  foreign  languages  and  the  type  of  operating  system  used  to  run  the  software 
received  scores  less  than  3.2  and  were  designated  as  least  important  (Table  6).  However, 
the  ability  to  support  foreign  languages  was  rated  higher  by  the  foreign  language  faculty, 
receiving  a  mean  rating  of  5.75  on  a  scale  of  6. 

Table  6:  The  mean  ratings  of  CMS  features  faculty  (n  =  29)  rated  significantly  higher  or 
lower  than  the  grand  mean  for  all  features.  The  corresponding  mean  scores  given  by 
cadets  (n=30)  are  also  shown.  Features  that  were  given  equivalent  ratings  by  both  groups 
are  shown  in  bold.  The  ratings  were  on  a  6  point  scale  where  1  indicated  that  the  feature 
was  of  very  little  importance  to  web-enabled  curriculum,  and  6  was  very  important  or  a 
“must  have”.  The  grand  mean  score  for  all  features  was  4.6  +  0.7  for  faculty  and  4.1  + 
0.82  for  cadets. 


Feature 

s.d. 

Cadet 

S.d. 

more  important: 

Course  can  be  downloaded  for  safekeepinq 

5.4 

0.8 

4.9 

1.2 

Can  use  a  mixture  of  question  types  on  a  sinqie  quiz 

5.4 

0.7 

4.7 

1.1 

Automated  link  to  course  material 

5.3 

5.2 

1.1 

Allows  links  to  custom  developed  pqs 

5.3 

0.7 

4.6 

1.5 

Allows  custom  qraphics 

5.3 

0.8 

4.4 

1.5 

Class  list  entered  one  at  at  time  or  uploaded  as  file 

5.3 

1.1 

4.6 

1.3 

Class  lists  can  be  presented,  saved,  printed  usinq  Excel 

5.3 

1.1 

4.7 

1.4 

Quizzes  automaticallv,  qraded  and  entered  into  qradebook 

5.3 

1 

4.9 

1.3 

Question  file  upload  capability 

5.3 

0.7 

4.1 

1.6 

less  important: 

Loqqed  synchronous  discussion 

3.8 

1.3 

3.3 

1.7 

Virtual  field  trips  within  discussion  paqes 

3.8 

1.4 

3.8 

1.4 

Adaptable  for  desktop  videoteleconferencinq 

3.8 

1.3 

3.7 

1.8 

Requires  pluq-ins 

3.5 

1.7 

3.4 

1 

Conforms  to  IMS  soecifications 

3.4 

1.9 

3.1 

1.5 

least  important: 

Support  foreiqn  lanquaqes 

3 

1.9 

3 

1.6 

Operatinq  System 

2 

1.7 

Eight  features  were  rated  5.0  or  higher  and  considered  to  be  more  important  than 
the  others  to  cadets  for  web-enabled  curriculum.  The  features  included  in  this  group  were 
the  ability  to  access  grades  and  navigational  features  such  as  a  search  tool,  online  student 
manual,  and  static  toolbar  (Table  7).  Six  features  received  a  score  of  3.2  or  below  and 
were  considered  less  important,  including  the  ability  to  conform  to  external  web 
development  specifications  such  as  IMS,  the  ability  to  support  foreign  languages,  and  the 
ability  to  track  students  access  to  the  materials.  The  operating  system  used  to  run  the 


software  received  a  score  of  2.4  and  was  determined  to  be  least  important  to  the  cadets 
(Table  7). 


Table  7:  The  mean  ratings  of  course  management  software  features  that  cadets  (n  =  30) 
rated  significantly  higher  or  lower  than  the  grand  mean  for  all  features.  The 
corresponding  mean  scores  given  by  faculty  (n=29)  are  also  shown.  Features  that  were 
given  equivalent  ratings  by  both  groups  are  shown  in  bold.  The  ratings  were  on  a  6  point 
scale  where  1  indicated  that  the  feature  was  of  very  little  importance  to  web-enabled 
curriculum,  and  6  was  very  important  or  a  “must  have”.  The  grand  mean  score  for  all 
features  was  4.6  +  0.70  for  faculty  and  4.1  +  0.82  for  cadets. 


Feature 

Cadet 

s.d. 

Facultv 

S.d. 

more  important 

Student  access  to  proqress  data  available 

5.6 

0.7 

5.0 

1.1 

Online  student  manual 

5.4 

0.7 

4.8 

1.2 

Search  tool  for  course  material 

5.4 

0.9 

5.1 

0.7 

Table  of  contents 

5.3 

0.8 

5.1 

1.3 

Automated  link  to  course  material  content 

5.2 

1.1 

5.3 

Instructor  comments  with  qrade 

5.1 

1.2 

5.0 

1.1 

Static  toolbar 

5.1 

1.1 

4.0 

1.8 

True-False/Multiple  choice  quiz  questions 

5.1 

1.3 

5.2 

0.9 

5.0 

1.3 

5.2 

0.8 

less  important: 

Conforms  to  IMS  soecifications 

3.1 

1.5 

3.4 

1.9 

Can  support  foreiqn  lanquaqes 

3.0 

1.6 

3.0 

1.9 

Can  track  how  often  student  accesses  paqes 

3.0 

1.5 

4.7 

1.2 

Can  track  time  spent  on  each  pace 

3.0 

1.5 

4.4 

1.1 

Student  trackinq  features 

3.0 

1.5 

5.0 

1.3 

Can  track  when  student  accesses  paqes 

2.9 

1.4 

4.6 

1.1 

Least  immportant 

Ooeratino  svstem  fexceotfor  NT  and  linux 

2.4 

1.7 

2.7 

2.3.6  Faculty  evaluation  of  the  software  for  course  development 

Ten  faculty  members  finished  evaluations  comparing  Courseinfo  to  WebCT,  and 
5  faculty  finished  evaluations  comparing  Courseinfo  to  Intralearn.  All  15  participants  in 
both  groups  preferred  Courseinfo  over  the  other  products  evaluated,  irrespective  of  which 
package  they  tried  first.  Given  the  unanimous  choice  of  Courseinfo  and  time  constraints 
of  the  faculty,  we  did  not  run  a  WebCT  vs  Intralearn  comparison  group  and  stopped 
recruiting  more  faculty  members  to  compare  Courseinfo  with  Intralearn  once  the 
minimum  sample  size  of  5  individuals  had  been  reached  (Nielsen,  2000).  In  addition, 
since  all  faculty  chose  the  same  product,  analyses  to  determine  if  preference,  ease  of  use 
and  time  on  task  was  dependent  upon  presentation  order  was  not  done. 

Of  the  1 5  faculty  members  who  finished  the  experiments,  three  chose  provide 
written  commentary  instead  of  completing  the  comparison  surveys.  Therefore,  their  data 
are  not  represented  here  for  time  on  task,  ease  of  use,  or  usefulness  of  the  features 
evaluated.  There  was  no  significant  difference  between  the  scores  given  to  Intralearn  and 


WebCT  for  features  they  shared  in  common,  so  those  data  have  been  pooled  to  simplify 
the  analyses. 


2.2.6a  Time  on  task 

It  took  faculty  1  hour  and  3 1  minutes  on  average  to  create  a  web-enabled  course 
using  Courseinfo  and  2  hours  and  22  minutes  on  average  to  create  a  course  identical  in 
content  using  either  WebCT  or  Intralearn  (Table  8).  However,  this  difference  was  not 
statistically  significant  due  to  inter-individual  differences  among  participants.  There  was 
no  significant  difference  in  the  amount  of  time  it  took  to  upload  assignments,  make 
quizzes,  do  simple  administrative  tasks  or  customize  the  look  and  feel  of  the  courses 
between  Courseinfo,  WebCT  and  Intralearn  ,  however  it  took  them  only  one-third  the 
time  to  set  up  a  course  using  Courseinfo  than  the  other  products. 

Table  8:  The  mean  amount  of  time  it  took  faculty  at  USAFA  (n=12)  to  create  a  web- 
enabled  course  using  one  of  three  CMS  software  packages.  Cl  =  Courseinfo  and  Other  = 
Intralearn  and  WebCT. 


Task 

Cl 

Other 

t 

df 

P 

SetuD 

0:18 

0:55 

2.20 

7 

0.063 

Assianments 

0:20 

0:31 

1.61 

8 

0.145 

Quizzes 

0:12 

0:15 

0.81 

9 

0.438 

Administration 

0:37 

0:40 

0.23 

6 

0.823 

Miscellaneous 

0:24 

0:30 

0.45 

6 

0.666 

Customization 

0:04 

0:11 

1.43 

3 

0.248 

Total 

1:31 

2:22 

1.54 

9 

0.158 

2.3.6b  Ease  of  Use 

Courseinfo  was  rated  significantly  easier  to  use  by  faculty  than  the  other  software 
packages  in  side-by-side  comparisons  for  16  of  the  18  tasks  evaluated  including  creating 
announcements,  composing  quizzes,  adding  assignments,  using  collaboration  features 
and  doing  administrative  tasks  (Table  9).  All  of  the  features  were  rated  as  somewhat  easy 
to  very  easy  with  the  exception  of  enrolling  the  cadets  in  class,  exporting  the  gradebook 
into  MS  Excel,  file  sharing  or  using  the  group  workspace,  which  were  rated  as  neutral. 
The  faculty  scored  nine  features  as  difficult  to  somewhat  difficult  and  none  of  the  features 
as  somewhat  easy  to  very  easy  for  WebCT  and  Intralearn  (Table  9).  However,  faculty 
members  did  not  use  the  online  manual  or  ask  for  help  significantly  more  often  when 
using  WebCT  or  Intralearn  than  they  did  when  using  Courseinfo  (Table  10). 


Table  9:  Comparison  of  faculty  (n  =  12)  mean  ease  of  use  seores  for  Courseinfo  (Cl)  and 
other  CMS  produets  (WebCT  and  Intralearn).  Anehors  for  this  seale  are  0  =  very 
diffieult,  12  =  diffieult,  29  =  somewhat  diffieult,  66  =  somewhat  easy,  82  =  easy,  100  = 
very  easy. 


Task 

Cl 

Other 

t 

df 

_ e _ 

Course  set  up 

86.9 

28.3 

8.49 

11 

o 

o 

o 

VI 

Addinq  assiqnments 

85.1 

37.3 

6.06 

11 

o 

o 

o 

VI 

Uploadinq  web  paqes 

79.2 

34.0 

4.22 

11 

CM 

o 

o 

o 

VI 

Linkinq  other  web  paqes 

74.1 

24.4 

4.18 

11 

CM 

O 

O 

O 

VI 

84.1 

43.5 

5.02 

11 

o 

o 

o 

VI 

Aministrative  functions  overall 

77.6 

31.9 

9.76 

11 

o 

o 

o 

VI 

Enrollinq  cadets 

56.1 

23.7 

4.96 

11 

o 

o 

o 

VI 

Creatinq  announcements 

81.2 

46.2 

4.13 

11 

CM 

o 

o 

o 

VI 

Sendinq  email  to  class 

83.3 

28.0 

7.19 

11 

o 

o 

o 

VI 

Settinq  up  qrade  book 

71.5 

15.4 

6.67 

11 

o 

o 

o 

VI 

Enterinq  qrades  into  qrade  book 

76.1 

22.9 

5.34 

11 

0.017 

Exportinq  qradebook  into  excel 

62.0 

18.0 

3.10 

11 

0.011 

Usinq  discussion  qroups 

72.1 

31.4 

3.20 

11 

lA 

O 

o 

o 

46.9 

26.8 

1.44 

11 

0.193 

Group  workspace 

40.9 

11.6 

2.82 

11 

o 

o 

o 

VI 

Calendar' 

77.9 

34.4 

2.89 

8 

0.020 

Chat  rooms 

71.6 

31.3 

4.01 

11 

0.003 

Modifying  look  and  feel  of  course' 

68.1 

45.3 

3.17 

8 

0.110 

'intralean  did  not  have  this  feature,  so  this  value  represents  a  eomparison  between 
Courseinfo  and  WebCT  only. 

Table  10:  Number  of  times  faeulty  (n  =  12)  partieipants  aeeessed  a  manual  or  asked  for 
help  when  using  a  one  of  three  eourse  management  software  produets.  Cl  =  Courseinfo 
and  Other  =  Intralearn  and  WebCT. 


Type  of  help 

Cl 

Other 

t 

df 

P 

Use  online  manual 

3.2 

3.4 

0.17 

9 

0.872 

Asked  questions 

0.2 

0.7 

1.25 

9 

0.244 

The  faeulty  also  rated  Courseinfo  as  being  the  most  eonsistent  with  how  other 
HTML  doeuments  were  displayed  when  uploaded  into  the  system.  All  of  the  paekages 
eaused  one  of  the  animated  pages  to  have  misaligned  graphies,  but  66%  of  the  pages 
appeared  the  same  before  and  after  they  were  uploaded  into  Courseinfo,  whereas  only 
50%  of  the  pages  appeared  the  same  when  uploaded  into  WebCT.  None  of  the 
partieipants  eould  find  any  graphies  on  the  pages  when  they  were  uploaded  into 
Intralearn. 


2.3.6c  Usefulness 

The  faculty  rated  the  tasks  they  were  asked  to  perform  using  Courseinfo 
significantly  more  useful  to  teaching  than  the  tasks  they  were  asked  to  complete  with  the 
other  software  packages  in  side-by-side  comparisons  for  17  of  the  18  tasks  evaluated, 
even  though  the  tasks  they  were  asked  to  do  with  each  product  were  identical.  These 
tasks  included  creating  announcements,  setting  up  the  grade  book,  and  uploading  web 
pages  created  using  other  software  products.  (Table  1 1).  All  of  the  features  were  rated  as 
somewhat  useful  to  very  useful  by  the  faculty  using  Courseinfo  with  the  exception  of 
discussion  groups,  the  group  workspace,  chat  rooms  and  modifying  the  look  and  feel  of 
the  course,  which  were  rated  as  neutral.  The  faculty  scored  none  of  the  18  tasks  as 
somewhat  useful  to  very  useful  for  WebCT  and  Intralearn,  but  scored  16  of  the  tasks  as 
neutral,  and  chat  room  and  discussion  groups  of  limited  use  (Table  1 1). 

Table  11:  Comparison  of  mean  usefulness  scores  given  by  faculty  (n  =  12)  for 
Courseinfo  (Cl)  and  other  CMS  products  (WebCT  and  Intralearn).  Anchors  for  this  scale 
are  0  =  not  at  all  useful,  12  =  not  very  useful,  29  =  of  limited  use,  66  =  somewhat  useful, 
82  =  useful,  100  =  very  useful. 


Task 

Cl 

Other 

t 

df 

P 

Course  set  up 

84.1 

50.1 

2.82 

11 

0.230 

Addinq  assiqnments 

87.5 

44.1 

3.18 

11 

0.015 

Uploadinq  web  paqes 

92.1 

35.7 

3.48 

11 

0.008 

Linkinq  other  web  paqes 

74.0 

36.2 

2.97 

11 

0.018 

88.1 

49.0 

3.05 

11 

0.019 

Administrative  functions  overaii 

80.4 

54.0 

2.78 

11 

0.024 

Enroiiinq  cadets 

76.8 

43.4 

2.71 

11 

0.027 

Creatinq  announcements 

83.6 

51.6 

2.89 

11 

0.018 

Sendinq  emaii  to  ciass 

80.1 

38.4 

3.32 

11 

0.011 

Settinq  up  qrade  book 

79.6 

44.6 

2.80 

11 

0.021 

Enterinq  qrades  into  qrade  book 

78.6 

51.6 

2.31 

11 

0.046 

Exportinq  qradebook  into  excei 

79.5 

48.3 

2.37 

11 

0.050 

Usinq  discussion  qroups 

53.7 

26.1 

2.44 

11 

0.040 

74.4 

39.0 

3.02 

11 

0.017 

Group  workspace 

63.3 

33.4 

3.01 

11 

0.017 

Caiendar' 

73.3 

36.8 

2.80 

8 

0.020 

Chat  rooms 

53.8 

25.0 

3.01 

11 

0.017 

Modifying  iook  and  feei  of  course' 

62.8 

29.9 

3.17 

8 

0.011 

'intralean  did  not  have  this  feature,  so  this  value  represents  a  comparison  between 
Courseinfo  and  WebCT  only. 

Since  the  tasks  they  were  asked  to  perform  with  each  CMS  package  were  the 
same,  faculty  should  have  given  them  the  same  scores,  therefore  it  appears  the  scores  that 
the  faculty  gave  the  tasks  were  more  dependent  upon  the  software  package  or  whether 
they  were  easy  to  do,  than  how  useful  these  tasks  were  to  teaching.  A  regression  analysis 
of  ease  of  use  and  usefulness  scores  assigned  for  both  software  packages  indicates  that 


scores  that  faculty  assigned  for  usefulness  were  significantly  dependent  upon  the  seores 
faeulty  assigned  for  ease  of  use  (r  =  0.8,  p  <  0.001,  df  =  35). 


2.3.6d  Overall  evaluation  by  faculty 

Courseinfo  was  rated  signifieantly  higher  than  the  other  software  packages  for  10 
statements  designed  to  ascertain  their  overall  experienee  using  the  software  produet. 
Faculty  felt  that  Courseinfo  had  a  signifioantly  better  navigational  layout,  was  more 
intuitive  in  nature  and  required  little  training  to  use,  as  well  as  had  pedagogieally  useful 
features.  All  statements  as  they  applied  to  Courseinfo  were  rated  as  moderately  agree  to 
decidedly  agree,  except  for  the  online  user’s  manual  and  the  direetions  given  on  the 
screen.  The  faeulty  scored  seven  of  the  10  statements  features  as  moderately  disagree  to 
decidedly  disagree  for  WebCT  and  Intralearn  ineluding  the  navigational  layout,  the 
pedagogieal  layout,  was  easy  to  use,  and  needs  little  training  to  use  (Table  12). 

Table  12:  Comparison  of  mean  overall  assessment  scores  by  faeulty  (n  =  12)  for 
Courseinfo  (Cl)  and  other  CMS  produets  (WebCT  and  Intralearn).  Anchors  for  this  scale 
are  0  =  decidedly  disagree,  25  =  moderately  disagree,  50  =  neutral,  15=  moderately 
agree,  100  =  deeidedly  agree. 


Statement 

Cl 

Other 

t 

df 

P 

Took  a  reasonable  amount  of  time 

76.4 

40.0 

2.45 

11 

0.034 

Had  a  qood  naviqational  layout 

80.0 

24.1 

4.43 

11 

0.001 

Had  a  qood  pedaqoqical  layout 

73.5 

24.0 

4.92 

11 

0.001 

75.0 

48.6 

2.97 

11 

0.015 

Was  easy  to  use 

83.6 

9.6 

11.20 

11 

0.001 

Had  an  easy  to  use  manual 

66.0 

18.5 

5.16 

11 

0.001 

Had  instructiye  directions  on  the  screen 

67.7 

23.2 

5.15 

11 

0.001 

Had  pedaqoqically  useful  features 

80.0 

40.0 

4.15 

11 

0.002 

Was  intuitiye  in  nature 

83.6 

12.7 

9.99 

11 

0.001 

Needs  little  traininq  to  use 

79.6 

9.1 

9.05 

11 

0.001 

2.3. 7  Evaluation  of  CMS  courses  by  cadets 

Twenty  nine  of  the  thirty  eadets  finished  the  evaluation  of  the  course  management 
software  produets.  Of  the  ten  eadets  who  evaluated  eourses  built  with  Courseinfo  and 
WebCT,  seven  preferred  eourses  built  with  Courseinfo  and  only  three  preferred  eourses 
built  with  WebCT.  Of  the  nine  eadets  who  evaluated  eourses  built  with  Courseinfo  and 
Intralearn,  all  nine  preferred  the  eourses  built  with  Courseinfo.  And  of  the  ten  cadets  who 
evaluated  eourses  built  with  Intralearn  and  WebCT,  five  preferred  eourses  built  with 
Intralearn  and  five  preferred  eourses  built  with  WebCT. 

There  was  no  preferenee  for  software  package  based  on  the  order  in  whieh  the 
produets  were  evaluated.  Cadets  ehose  the  first  package  that  they  were  presented  44.8% 
and  the  seeond  paekage  they  were  presented  55.2%  of  the  time.  Since  the  eourses  were 
presented  to  the  cadets  aeeording  to  a  randomization  scheme,  eaeh  course  had  an  equal 
ehance  of  being  presented  first  or  second. 


To  evaluate  the  courses,  cadets  located  announcements  and  the  syllabus, 
submitted  assignments,  accessed  external  links,  used  the  discussion  boards,  took  quizzes 
and  checked  their  grades.  Cadets  were  given  the  answers  to  the  quizzes,  and  the  quizzes, 
syllabus  and  supporting  materials  were  identical  for  each  course,  however  each  course 
had  a  different  navigational  interface.  The  interface  for  Courseinfo  and  WebCT  required 
the  cadets  to  search  for  assignments  for  a  particular  lesson  under  different  course 
headings,  whereas  the  navigational  layout  for  Intraleam  had  the  daily  lessons  along  with 
the  assignments  listed  on  the  opening  page. 


2.3. 7a  Time  on  task 

Cadets  spent  an  average  of  26.6  minutes  evaluating  each  course.  There  was  no 
significant  difference  for  the  overall  amount  of  time  that  it  took  them  to  complete  the 
tasks  for  each  course  (Table  13  a-c),  but  it  took  the  cadets  significantly  less  time  to 
complete  quizzes  in  the  course  built  with  Courseinfo  than  WebCT,  and  significantly  less 
time  to  find  the  syllabus  and  supporting  documents  in  the  course  built  with  Courseinfo 
than  with  Intraleam.  Cadets  spent  statistically  significant  less  time  completing  the 
evaluation  on  the  second  software  package  than  they  did  the  first  software  package 
(Table  14). 

Table  13a:  The  mean  amount  of  time  in  minutes  it  took  cadets  at  USAFA  (n=10)  to 
navigate  and  access  materials  in  web-enabled  courses  created  with  Cl  (Courseinfo)  and 
WebCT. 


Task 

WebCT 

Cl 

t 

df 

P 

Locatinq  course  documents 

6.5 

6.5 

0.00 

8 

1.00 

Completinq  assiqnments 

8.5 

7.4 

0.40 

8 

0.60 

Accessinq  external  links 

1.1 

0.9 

1.31 

8 

0.22 

Usinq  communication  tools 

6.2 

6.2 

0.00 

8 

1.00 

Takinq  quizzes 

7.0 

5.0 

3.32 

8 

0.01 

Total  time 

26.5 

23.5 

0.77 

9 

0.45 

Table  13b:  The  mean  amount  of  time  in  minutes  it  took  cadets  at  USAFA  (n=9)  to 
navigate  and  access  materials  in  web-enabled  courses  created  with  Cl  (Courseinfo)  and 
Intraleam. 


Task 

Cl 

Intraleam 

t 

df 

P 

Locatinq  course  documents 

4.8 

7.8 

3.00 

8 

0.01 

Completinq  assiqnments 

5.5 

6.9 

0.70 

7 

0.49 

Accessinq  external  links 

0.9 

1.8 

1.45 

8 

0.18 

Usinq  communication  tools 

5.3 

5.1 

0.21 

8 

0.83 

Takinq  quizzes 

5.1 

5.4 

0.10 

7 

0.91 

Total  time 

20.6 

26.9 

1.37 

8 

0.20 

Table  13c:  The  mean  amount  of  time  in  minutes  it  took  cadets  at  USAFA  (n=10)  to 
navigate  and  access  materials  in  web-enabled  courses  created  with  WebCT  and 
Intralearn. 


Task 

WebCT 

Intralearn 

t 

df 

P 

Locatinq  course  documents 

6.5 

9.2 

1.71 

9 

0.11 

Completinq  assiqnments 

11.0 

8.6 

0.64 

9 

0.53 

Accessinq  external  links 

1.5 

2.2 

1.73 

9 

0.11 

Usinq  communication  tools 

4.7 

7.9 

1.12 

9 

0.29 

Takinq  quizzes 

5.6 

4.3 

1.74 

8 

0.11 

Total  time 

30.4 

31.8 

0.17 

9 

0.86 

Table  14:  The  mean  amount  of  time  in  minutes  it  took  cadets  at  USAFA  (n=29)  to 
navigate  and  access  materials  in  web-enabled  courses  created  with  CMS  software  by 
order  in  which  the  course  was  presented  to  the  students. 


Task 

first 

second 

t 

df 

P 

Locatinq  course  documents 

8.0 

5.9 

2.74 

27 

0.01 

Completinq  assiqnments 

9.4 

6.9 

1.63 

26 

0.11 

Accessinq  external  links 

1.5 

1.3 

0.90 

27 

0.37 

Usinq  communication  tools 

6.6 

5.3 

1.06 

27 

0.29 

Takinq  quizzes 

6.3 

4.8 

1.20 

26 

0.23 

Total  time 

30.2 

22.3 

2.25 

28 

0.03 

2.3. 7b  Ease  of  Use 

There  was  no  significant  difference  between  courses  built  with  Intralearn  and 
WebCT,  or  between  courses  built  with  Courseinfo  and  WebCT  for  ease  of  use  scores  that 
cadets  assigned  to  12  different  tasks  (Tables  15a  and  c).  Cadets  rated  courses  built  with 
Courseinfo  significantly  easier  than  Intralearn  for  5  of  the  12  tasks  evaluated  (Table  15b). 
The  cadets  rated  most  of  the  features  as  somewhat  easy  to  very  easy  to  use.  The  only 
features  that  were  found  difficult  for  all  of  the  courses  was  using  the  drop  box  to  submit 
assignments  to  the  instructor.  Cadets  also  rated  using  the  discussion  boards  in  the  WebCT 
course  somewhat  difficult,  as  well  as  locating  announcements,  checking  their  grades  and 
using  the  calendaring  function  in  the  Intralearn  course,  (note:  Intralearn  does  not  have  a 
calendar  function). 


Table  15a:  Comparison  of  cadet  (n  =  10)  mean  ease  of  use  scores  for  courses  built  with 
Intralearn  and  WebCT.  Anehors  for  this  seale  are  0  =  very  diffieult,  12  =  diffieult,  29  = 
somewhat  diffieult,  66  =  somewhat  easy,  82  =  easy,  100  =  very  easy. 


Task 

Intralearn 

WebCT 

t 

df 

P 

Locatinq  the  syllabus 

74.8 

82.2 

0.66 

9 

0.524 

Locatinq  the  announcements 

36.9 

66.7 

2.13 

6 

0.077 

Locatinq  instructor  information 

89.2 

89.4 

0.08 

9 

0.939 

Locatinq  the  course  policies 

90.1 

89.9 

0.08 

9 

0.939 

Locatinq  the  assiqnments 

79.6 

78.9 

0.16 

9 

0.876 

Usinq  the  drop  box 

26.0 

37.6 

0.83 

6 

0.437 

Locatinq  external  links 

68.2 

89.9 

1.93 

9 

0.086 

Usinq  the  discussion  boards 

64.4 

63.6 

0.10 

9 

0.922 

Usinq  the  calendar  function 

27.6 

87.4 

3.10 

4 

0.036 

Takinq  quizzes 

60.5 

78.0 

2.07 

7 

0.077 

Checkinq  yourqrade 

53.6 

73.4 

1.41 

7 

0.201 

Emailinq  the  instructor 

80.9 

76.6 

0.28 

8 

0.784 

Table  15b:  Comparison  of  eadet  (n  =  9)  mean  ease  of  use  seores  for  eourses  built  with 
Courseinfo  (Cl)  and  Intralearn.  Anehors  for  this  seale  are  0  =  very  diffieult,  12  = 
diffieult,  29  =  somewhat  difficult,  66  =  somewhat  easy,  82  =  easy,  100  =  very  easy. 


Task 

Cl 

Intralearn 

t 

df 

P 

Locatinq  the  syllabus 

89.0 

85.6 

0.96 

8 

0.364 

Locatinq  the  announcements 

94.4 

55.7 

2.49 

6 

0.047 

Locatinq  instructor  information 

90.7 

80.3 

2.13 

8 

0.066 

Locatinq  the  course  policies 

91.0 

81.0 

1.39 

8 

0.202 

Locatinq  the  assiqnments 

94.8 

86.7 

1.44 

8 

0.188 

Usinq  the  drop  box 

69.5 

32.3 

2.38 

7 

0.049 

Locatinq  external  links 

95.8 

70.9 

2.73 

8 

0.026 

Usinq  the  discussion  boards 

95.6 

77.4 

2.55 

8 

0.034 

Usinq  the  calendar  function 

90.0 

5.3 

13.83 

3 

0.001 

Takinq  quizzes 

93.8 

80.8 

1.51 

8 

0.170 

Checkinq  yourqrade 

82.5 

55.4 

1.18 

7 

0.277 

Emailinq  the  instructor 

94.4 

66.6 

2.12 

8 

0.067 

Table  15c:  Comparison  of  cadet  (n  =  10)  mean  ease  of  use  seores  for  eourses  built  with 
Courseinfo  (Cl)  and  WebCT.  Anehors  for  this  seale  are  0  =  very  diffieult,  12  =  diffieult, 
29  =  somewhat  diffieult,  66  =  somewhat  easy,  82  =  easy,  100  =  very  easy. 


Task 

Cl 

WebCT 

t 

df 

P 

Locatinq  the  syllabus 

76.7 

82.4 

0.56 

9 

0.588 

Locatinq  the  announcements 

94.5 

92.8 

0.41 

9 

0.692 

Locatinq  instructor  information 

93.5 

93.7 

1.00 

9 

0.343 

Locatinq  the  course  policies 

92.7 

94.2 

0.71 

9 

0.496 

Locatinq  the  assiqnments 

94.7 

90.2 

1.71 

9 

0.121 

Usinq  the  drop  box 

54.2 

46.9 

0.52 

9 

0.619 

Locatinq  external  links 

94.5 

96.0 

0.60 

9 

0.566 

Usinq  the  discussion  boards 

74.1 

57.6 

1.52 

9 

0.163 

Usinq  the  calendar  function 

90.0 

89.8 

0.07 

9 

0.945 

Takinq  quizzes 

84.7 

81.4 

1.12 

9 

0.294 

Checkinq  yourqrade 

86.3 

80.2 

1.19 

8 

0.268 

Emailinq  the  instructor 

93.9 

88.3 

1.25 

8 

0.247 

Cadets  rated  the  seeond  eourse  they  evaluated  as  slightly  easier  to  use  in  nine  of 
the  12  me  tries  examined.  However,  this  differenee  was  not  statistieally  signifieant  (Table 
16). 


Table  16:  Comparison  of  eadet  (n  =  29)  mean  ease  of  use  seores  for  the  web-enabled 
eourses  they  evaluated  first  and  seeond.  Anehors  for  this  seale  are  0  =  very  diffieult, 

12  =  diffieult,  29  =  somewhat  diffieult,  66  =  somewhat  easy,  82  =  easy,  100  =  very  easy. 


Task 

First 

Second 

t 

df 

P 

Locating  the  syllabus 

88.4 

92.6 

1.47 

28 

0.15 

Locating  the  announcements 

75.3 

77.2 

0.23 

24 

0.81 

Locating  instructor  information 

85.0 

86.0 

0.76 

28 

0.45 

Locating  the  course  policies 

82.2 

84.9 

1.4 

28 

0.17 

Locating  the  assignments 

89.9 

91.6 

0.89 

28 

0.37 

Using  the  drop  box 

49.3 

60.3 

1.41 

28 

0.16 

Locating  external  links 

87.1 

83.6 

0.68 

28 

0.5 

Using  the  discussion  boards 

68.2 

64.5 

1.33 

28 

0.19 

Using  the  calendar  function 

77.4 

67.7 

0.92 

22 

0.36 

Taking  quizzes 

80.1 

81.9 

0.4 

26 

0.68 

Checking  your  grade 

69.7 

78.6 

0.94 

27 

0.35 

Emailing  the  instructor 

72.7 

81.9 

1.65 

28 

0.1 

2.3.7c  Usefulness: 

There  was  no  signifieant  differenee  between  the  usefulness  of  the  tasks  eadets 
performed  in  eourses  built  with  Intralearn  and  WebCT  for  1 1  of  the  12  tasks,  between 
eourses  built  with  Courseinfo  and  Intralearn  for  9  of  12  tasks,  or  between  eourses  built 
with  Courseinfo  and  WebCT  for  11  of  12  tasks  (Tables  17a-e).  As  with  the  faeulty  the 
tasks  that  they  were  asked  to  evaluate  were  the  same  for  eaeh  of  the  web-enabled  eourses. 
The  eadets  rated  most  of  the  features  as  somewhat  useful  to  very  useful.  The  only  features 


that  were  rated  neutral  were  using  the  drop  box  for  submitting  assignments  to  the 
instruetor  and  the  diseussion  boards  for  ail  eourses.  Cadets  also  rated  eheeking  their 
grades  and  using  the  ealendaring  funetion  in  Intraleam  as  neutral. 


Table  17a:  Comparison  of  cadet  (n  =  10)  mean  usefulness  scores  for  tasks  evaluated  in 
web-enabled  courses  built  with  Intraleam  and  WebCT.  Anchors  for  this  scale  are  0  =  not 
at  all  useful,  12  =  not  very  useful,  29  =  of  limited  use,  66  =  somewhat  useful,  82  =  useful, 
100  =  very  useful. 


Task 

Intraleam 

WebCT 

t 

df 

P 

Locatinq  the  syllabus 

92.5 

95.5 

0.59 

9 

0.572 

Locatinq  the  announcements 

53.0 

68.0 

1.25 

6 

0.258 

Locatinq  instructor  information 

78.2 

79.2 

1.00 

9 

0.343 

Locatinq  the  course  policies 

76.8 

76.8 

0.00 

9 

0.000 

Locatinq  the  assiqnments 

88.5 

91.0 

1.00 

9 

0.343 

Usinq  the  drop  box 

36.8 

63.3 

2.37 

9 

0.042 

Locatinq  external  links 

65.1 

82.6 

1.56 

9 

0.152 

Usinq  the  discussion  boards 

49.2 

52.2 

0.54 

9 

0.604 

Usinq  the  calendar  function 

51.3 

85.5 

2.28 

7 

0.057 

Takinq  quizzes 

78.1 

85.0 

1.46 

7 

0.188 

Checkinq  yourqrade 

55.6 

78.3 

1.74 

8 

0.120 

Emailinq  the  instructor 

77.2 

67.7 

0.94 

9 

0.370 

Table  17b:  Comparison  of  cadet  (n  =  9)  mean  usefulness  scores  for  tasks  evaluated  in 
web-enabled  courses  built  with  Courseinfo  (Cl)  and  Intraleam.  Anchors  for  this  scale  are 
0  =  not  at  all  useful,  12  =  not  very  useful,  29  =  of  limited  use,  66  =  somewhat  useful,  82 
=  useful,  100  =  very  useful. 


Task 

Cl 

Intraleam 

t 

df 

P 

Locatinq  the  syllabus 

95.4 

89.7 

1.31 

8 

0.227 

Locatinq  the  announcements 

98.6 

56.3 

2.57 

7 

0.037 

Locatinq  instructor  information 

93.9 

85.6 

1.86 

8 

0.100 

Locatinq  the  course  policies 

86.1 

82.3 

0.59 

8 

0.571 

Locatinq  the  assiqnments 

85.6 

89.4 

0.63 

8 

0.546 

Usinq  the  drop  box 

80.6 

37.7 

2.86 

8 

0.021 

Locatinq  external  links 

97.1 

78.9 

2.11 

8 

0.067 

Usinq  the  discussion  boards 

86.6 

84.2 

0.32 

8 

0.760 

Usinq  the  calendar  function 

95.0 

33.4 

3.22 

4 

0.032 

Takinq  quizzes 

95.6 

90.0 

1.19 

8 

0.269 

Checkinq  yourqrade 

84.6 

56.2 

1.39 

8 

0.201 

Emailinq  the  instructor 

95.1 

72.3 

1.87 

8 

0.098 

Table  17c:  Comparison  of  cadet  (n  =  10)  mean  usefulness  seores  for  tasks  evaluated  in 
web-enabled  eourses  built  with  Courseinfo  (Cl)  and  WebCT.  Anehors  for  this  seale  are  0 
=  not  at  all  useful,  12  =  not  very  useful,  29  =  of  limited  use,  66  =  somewhat  useful,  82  = 
useful,  100  =  very  useful. 


Task 

Cl 

WebCT 

t 

df 

P 

Locatinq  the  syllabus 

86.1 

89.9 

0.59 

9 

0.572 

Locatinq  the  announcements 

87.7 

87.7 

0.00 

9 

0.000 

Locatinq  instructor  information 

91.1 

88.9 

1.49 

9 

0.170 

Locatinq  the  course  policies 

88.9 

85.9 

1.41 

9 

0.193 

Locatinq  the  assiqnments 

92.4 

88.7 

1.14 

9 

0.285 

Usinq  the  drop  box 

57.7 

60.6 

0.35 

9 

0.733 

Locatinq  external  links 

96.5 

93.7 

1.43 

9 

0.187 

Usinq  the  discussion  boards 

65.7 

55.8 

2.93 

9 

0.017 

Usinq  the  calendar  function 

74.9 

86.7 

1.19 

9 

0.267 

Takinq  quizzes 

67.8 

73.7 

0.55 

9 

0.599 

Checkinq  yourqrade 

82.0 

86.0 

0.35 

9 

0.733 

Emailinq  the  instructor 

78.7 

74.7 

1.00 

9 

0.343 

2.3. 7d  Overall  evaluation  by  cadets 

There  was  no  signifieant  differenee  between  eourses  built  with  Intraleam  and 
WebCT  for  10  statements  designed  to  aseertain  eadet’s  overall  experienee  (Table  18a). 
Most  of  the  statements  were  given  a  rating  of  neutral. 

Table  18a:  Comparison  of  mean  overall  assessment  seores  by  eadets  (n  =  10)  for  eourses 
built  with  Intraleam  and  WebCT.  Anehors  for  this  seale  are  0  =  deeidedly  disagree,  25  = 
moderately  disagree,  50  =  neutral,  75  =  moderately  agree,  100  =  deeidedly  agree. 


This  software  package: 

Intraleam 

WebCT 

t 

df 

P 

Took  a  reasonable  amount  of  time 

62.0 

59.0 

0.33 

9 

0.749 

Had  a  qood  naviqational  layout 

48.5 

62.0 

0.89 

9 

0.398 

Had  a  loqical  layout 

57.0 

70.0 

0.89 

9 

0.394 

63.0 

71.5 

0.52 

9 

0.618 

Was  easy  to  use 

56.0 

67.5 

0.91 

9 

0.387 

Had  an  easy  to  use  manual 

27.5 

52.5 

1.67 

1 

0.344 

Had  instructiye  directions  on  the  screen 

47.2 

47.2 

0.00 

9 

1.000 

Had  useful  features 

69.5 

79.0 

1.11 

9 

0.297 

Was  intuitiye  in  nature 

52.2 

72.8 

1.91 

8 

0.093 

Needs  little  traininq  to  use 

69.0 

68.0 

0.09 

9 

0.931 

The  web-enabled  eourse  built  with  Courseinfo  was  rated  signifieantly  higher  than 
the  eourse  built  with  Intraleam  for  4  of  the  10  statements  used  to  assess  the  eadet’s 
overall  experienee  (Table  18b).  Cadets  rated  9  of  the  10  statements  as  moderately  agree 
to  decidedly  agree  as  they  pertain  to  the  Courseinfo  eourse  and  only  one  of  the 
statements  was  ranked  as  moderately  agree  for  the  Intraleam  eourse.  The  other 
statements  were  rated  as  neutral. 


Table  18b:  Pairwise  comparison  of  overall  assessment  statements  by  cadets  (n  =  9)  for 
courses  built  with  Courseinfo  (Cl)  and  Intralearn.  Anchors  for  this  scale  are  0  = 
decidedly  disagree,  25  =  moderately  disagree,  50  =  neutral,  15=  moderately  agree,  100  = 
decidedly  agree. 


This  software  package: 

Cl 

Intralearn 

t 

df 

P 

Took  a  reasonable  amount  of  time 

98.1 

71.5 

3.58 

7 

0.009 

Had  a  qood  naviqational  layout 

90.6 

45.6 

5.50 

7 

0.001 

Had  a  loqical  layout 

92.5 

55.4 

4.96 

7 

0.002 

87.5 

72.5 

1.98 

7 

0.088 

Was  easy  to  use 

89.9 

55.9 

3.72 

7 

0.007 

Had  an  easy  to  use  manual 

73.8 

48.8 

1.89 

3 

0.155 

Had  instructiye  directions  on  the  screen 

70.0 

39.3 

2.33 

6 

0.058 

Had  useful  features 

95.6 

78.1 

2.15 

7 

0.068 

Was  intuitiye  in  nature 

93.8 

68.5 

3.56 

7 

0.009 

Needs  little  traininq  to  use 

83.8 

62.5 

1.66 

7 

0.141 

The  course  built  with  Courseinfo  was  rated  significantly  higher  than  the  courses 
built  with  WebCT  for  1  of  the  10  statements  used  to  assess  the  cadet’s  overall  experience 
(Table  18c).  However,  the  cadets  rated  9  of  the  10  statements  as  moderately  agree  to 
decidedly  agree  as  they  pertain  to  Courseinfo  and  only  one  of  the  statements  was  ranked 
as  moderately  agree  for  WebCT.  The  other  statements  were  rated  as  neutral. 

Table  18c:  Comparison  of  mean  overall  assessment  scores  by  cadets  (n  =  10)  for  courses 
built  with  Courseinfo  (Cl)  and  WebCT.  Anchors  for  this  scale  are  0  =  decidedly  disagree, 
25  =  moderately  disagree,  50  =  neutral,  75  =  moderately  agree,  100  =  decidedly  agree. 


This  software  package: 

Cl 

WebCT 

t 

df 

P 

Took  a  reasonable  amount  of  time 

84.0 

72.5 

1.13 

9 

0.288 

Had  a  qood  nayiqational  layout 

90.0 

73.0 

1.60 

9 

0.144 

Had  a  loqical  layout 

87.0 

77.0 

0.97 

9 

0.358 

88.0 

61.0 

2.29 

9 

0.048 

Was  easy  to  use 

86.0 

69.0 

1.63 

9 

0.136 

Had  an  easy  to  use  manual 

91.0 

62.0 

1.99 

4 

0.118 

Had  instructiye  directions  on  the  screen 

60.0 

43.1 

1.86 

7 

0.105 

Had  useful  features 

89.8 

74.0 

1.57 

9 

0.151 

Was  intuitiye  in  nature 

82.5 

71.5 

0.91 

9 

0.385 

Needs  little  traininq  to  use 

81.0 

68.0 

1.07 

9 

0.311 

Cadets  found  information  where  they  expected  to  find  it  less  that  50%  of  the  time 
in  any  of  the  courses  built  using  any  of  the  CMS  products  evaluated  in  this  series  of 
experiments  (Table  19).  Cadets  also  reported  finding  course  documents  and  other 
information  in  places  where  it  was  not  located.  Similarly,  when  faculty  were  asked  to 
place  documents  into  a  course  using  one  of  these  products,  different  faculty  members  put 
the  documents  in  different  places  (Table  20). 


Table  19:  The  locations  within  the  course  where  cadets  expected  to  find  and  did  find  the 
instructor’s  office  hours  and  two  course  handouts  in  a  web-enabled  course  built  with  one 
of  three  CMS  products.  The  correct  location  of  the  document  or  information  is  shown  in 
bold  face  font. 


Syllabus 

Office  location 

Honors  document  | 

BB 

exp. 

found 

exp. 

found 

exp. 

found 

Course  Information 

6 

13 

1 

7 

4 

Staff  Information 

14 

19 

Course  Documents 

3 

11 

14 

Other 

10(5) 

6(3) 

4(2) 

2(2) 

WebCT 

Instructor  Information 

11 

20 

Course  Documents 

11 

16 

15 

14 

Other 

9(3) 

5(3) 

9(4) 

5(2) 

6(2) 

Intralearn 

Syiiabus 

8 

7 

9 

17 

4 

3 

Roster 

2 

Communication 

1 

Other 

11(3) 

15(3) 

3(3) 

2(2) 

15(6) 

17(5) 

Table  20:  The  locations  where  faculty  placed  two  handouts  into  a  course  they  were 
building  using  a  CMS  product. 


Syllabus 

Honors 

BB  Courseinfo 

Course  Information 

8 

4 

Course  Documents 

6 

7 

Course  Materiais 

1 

Academics  w/honors  Cisrm  poi 

1 

WebCT 

Course  Documents 

Course  Syiiabus/Course  Materiais 

6 

3 

Fiie  Manager 

1 

1 

N/A 

3 

5 

Homepage 

1 

Intralearn 

Syiiabus 

N/A 

3 

2 

Reference  Materiais 

1 

Upioads 

1 

1 

2.4  Discussion 


2.4.1  WWW  experience  by  faculty  and  cadets 

The  differences  between  the  experience  of  faculty  and  cadets  in  the  number  of 
years  they  used  and  developed  content  for  the  www,  was  not  significantly  different,  even 
though  faculty  are  more  educated  and  have  more  computer  experience  than  cadets.  Both 
groups  over-estimated  the  number  of  years  they  used  web  browsers,  as  many  reported 
using  these  tools  before  they  had  been  developed  (Anderberg,  2001;  Stewart,  2000). 
While  it  is  possible  that  some  participants  may  have  used  this  technology  in  its  early 
development  and  blended  various  aspects  of  the  internet  with  the  www  in  their  answers, 
it  is  more  likely  this  over-reporting  demonstrates  that  the  use  of  the  www  is  no  longer  a 
novel  concept  and  both  groups  are  very  comfortable  using  this  medium. 

Cadets  reported  using  the  web  twice  as  many  hours  per  week  as  faculty.  While 
faculty  reported  more  computer  programming  experience  overall,  cadets  were  just  as 
familiar  as  faculty  with  authoring  web  content,  and  cadets  were  just  as  likely  as  faculty  to 
host  their  own  web  site  on  a  server  located  off  the  academy.  However,  it  appears  that 
neither  group  uses  sophisticated  development  software  or  applications  for  their  web  sites, 
which  is  typical  of  most  personal  home  pages  on  the  www  (NetKontoret,  2001). 

2.4.2  Selection  of  CMS  features  by  both  faculty  and  cadets 

When  inexperienced  participants  were  asked  to  determine  what  features  were 
important  and  should  be  included  in  software  to  facilitate  the  development  and  use  of 
web-enabled  courses,  both  the  faculty  and  cadets  focused  on  their  own  use  of  the 
software  and  did  not  appear  to  consider  the  requirements  of  other  user  groups.  Faculty 
did  not  rate  features  that  were  to  be  used  by  cadets  such  as  finding  grades  or  the  ability  to 
navigate  the  site  higher  than  average.  The  faculty  focused  on  their  requirements  for  the 
tool,  such  as  the  uploading  of  class  lists  and  the  flexibility  to  add  content  from  other 
resources.  Both  faculty  and  cadets  rated  the  ability  to  support  foreign  languages  as  one  of 
the  least  important  features  of  these  tools,  yet  the  ability  to  support  languages  other  than 
English  would  be  essential  for  developing  web-enabled  curriculum  for  a  foreign  language 
course.  The  2  foreign  language  instructors  and  the  3  cadets  enrolled  in  foreign  language 
courses  rated  this  feature  as  a  “must-have”. 

Cadets  also  focused  on  their  own  requirements,  rather  than  the  development  of  the 
course  as  a  whole.  They  rated  the  exam  question  types  they  favored  and  the  ability  to  find 
materials  as  more  important  than  other  features.  Cadets  rated  student  tracking  features, 
such  as  the  ability  to  tell  when  and  for  how  long  a  student  accessed  a  page  as  less 
important  than  most  of  the  other  features  listed,  even  though  these  resources  could  be 
used  by  instructors  to  help  determine  which  students  were  at  risk  of  not  being  adequately 
prepared  for  an  exam  or  class.  Since  neither  group  seems  to  have  considered  other  user’s 
requirements,  this  illustrates  the  need  to  have  all  user  groups  involved  in  the  selection 
process  and  development  of  requirements  for  these  types  of  tools. 

Neither  faculty  nor  cadets  rated  the  collaboration  tools,  asynchronous  or 
synchronous  discussion  groups,  or  email  functions  as  being  very  important  to  include  in 


the  software  packages.  Unlike  most  colleges  and  universities,  class  attendance  is 
mandatory  for  all  cadets  at  USAFA,  and  faculty  are  generally  available  from  7:30  to  4:30 
every  day  outside  of  class  time.  Therefore,  the  ability  to  have  online  discussions,  or  to 
collaborate  on  projects  at  a  distance  may  not  be  as  important  as  it  is  for  those  using  these 
packages  for  true  distance  learning  courses,  where  faculty  and  students  are  separated  both 
temporally  and  spatially.  In  addition,  all  faculty  and  cadets  have  institutional  email 
accounts,  and  all  email  addresses  are  listed  in  a  centralized  address  book.  Therefore,  this 
feature  within  a  course  management  package  is  redundant  with  a  system  already  in  place 
and  in  use  at  USAFA. 

2.4.3  Selection  of preferred  software  package  by  faculty 

The  faculty  unanimously  selected  Blackboard  Courseinfo  as  the  preferred  CMS 
package  of  the  packages  they  tested  for  developing  web-enabled  courses.  All  of  the 
faculty  surveyed  indicated  that  they  chose  this  package  because  it  was  the  easiest  to  use, 
and  most  cited  ease  of  use  as  the  single  most  important  factor  that  helped  them  make  their 
decision.  They  indicated  that  the  navigational  interface  was  intuitive,  it  required  fewer 
mouse-clicks,  and  the  steps  necessary  to  load  assignments  and  course  documents  were 
obvious  in  comparison  to  WebCT  and  Intralearn. 

Even  though  the  tasks  that  they  were  asked  to  perform  were  the  same,  the  faculty 
rated  the  course  development  tasks  significantly  more  useful  when  using  Courseinfo  than 
with  either  WebCT  or  Intralearn.  There  was  a  strong  statistical  correlation  between  ease 
of  use  and  usefulness  scores  for  these  tasks.  If  a  task  was  easy  to  perform,  the  faculty 
tended  to  rate  it  as  more  useful  to  teaching  and  learning,  and  if  the  task  was  more  difficult 
to  perform,  then  the  faculty  rated  it  as  less  useful  to  teaching  and  learning.  Therefore, 
even  if  WebCT  and  Intralearn  had  more  useful  tools  than  Courseinfo,  they  probably 
would  still  be  rated  as  a  less  useful  product  overall. 

One  of  the  design  features  that  made  Courseinfo  easier  to  use  than  WebCT  and 
Intralearn  was  the  layout  of  the  menus  that  faculty  encountered  when  using  the  software 
to  build  a  course.  The  Courseinfo  menu  allows  the  user  to  view  the  different  subheadings 
available  and  choose  the  desired  function,  whereas  the  WebCT  interface  only  showed  a 
few  choices  with  each  screen.  In  order  to  figure  out  what  features  might  be  available 
under  a  certain  section,  the  faculty  member  had  to  click  and  wait  for  the  next  page  to 
load.  Intralearn  also  used  a  series  of  menus,  however  that  package  required  the  faculty 
member  to  upload  all  of  their  documents  into  the  system  first,  and  then  go  back  and 
integrate  it  into  the  content.  All  of  the  faculty  who  tried  to  use  Intralearn  reported  that 
once  they  uploaded  a  document  into  the  system,  they  could  not  locate  it. 

The  faculty  indicated  that  Courseinfo  did  not  require  a  lot  of  training,  whereas 
WebCT  and  Intralearn  would  require  a  lot  of  training  to  use.  However,  faculty  did  not 
seek  out  help  or  use  the  online  manuals  very  often  when  using  any  of  the  software 
packages.  One  reason  may  be  that  these  courses  were  not  going  to  be  used  outside  of  this 
evaluation  process.  Therefore  if  tasks  were  difficult  to  understand  or  execute,  faculty 
could  skip  that  section  and  proceed  to  the  next  set.  There  was  no  difference  in  computer 
use  or  programming  experience  between  the  faculty  who  finished  the  experiments  and  the 


faculty  who  did  not  finish,  which  indicates  that  both  groups  had  the  neeessary  skills  to 
finish  the  evaluation  proeess.  Most  of  the  partieipants  who  did  not  finish,  indieated  that 
they  were  too  busy,  yet  a  few  expressed  a  dislike  of  CMS  tools  overall  and  may  have 
been  biased  beeause  they  did  not  like  the  first  paekage  that  they  tried.  As  one  participant 
stated  “the  first  software  package  was  so  distasteful,  you  eould  not  eonvinee  me  to  try  a 
seeond.” 


Courseinfo  also  included  six  out  of  the  seven  features  that  faeulty  rated  as  more 
important  before  they  began  evaluating  the  software  packages.  The  only  feature  that  was 
not  present  was  the  ability  to  upload  questions  as  a  question  file,  and  they  had  not  been 
asked  to  do  this  task  as  part  of  the  evaluation  proeess.  Finding  features  that  they  had 
predetermined  as  useful  may  have  validated  their  beliefs  that  this  produet  was  benefieial 
to  teaching. 

2.4.4  Selection  of  preferred  software  package  by  cadets 

Cadets  who  eompared  eourses  prepared  with  Courseinfo  to  Intralearn, 
unanimously  ehose  the  eourse  designed  with  Courseinfo.  Although  three  of  the  cadets 
preferred  the  eourse  designed  with  WebCT  to  the  Courseinfo  eourse,  the  other  seven 
preferred  the  eourse  designed  with  Courseinfo.  Cadets  were  equally  divided  when  given 
a  ehoiee  between  courses  designed  with  WebCT  and  Intralearn. 

Overall  the  eadets  found  all  three  paekages  easy  to  use,  and  only  34%  of  the 
eadets  used  the  online  manuals  or  asked  for  help  during  the  proeess.  Cadets  did  indieate 
that  the  seeond  paekage  that  they  tried  was  easier  to  use  on  average  than  the  first  paekage 
that  they  tried.  This  was  probably  a  function  of  habituating  to  the  tasks  and  to  the 
worksheets,  than  the  aetual  software  paekages.  The  software  was  assigned  to  the  eadets  in 
a  randomized  fashion  so  that  eaeh  software  paekage  had  an  equal  probability  of  being 
first  or  seeond.  However,  this  was  not  in  and  of  itself  a  deciding  factor  since  eadets  did 
not  show  a  statistieal  bias  towards  ehoosing  the  seeond  eourse  as  their  overall  preferred 
ehoiee. 


The  eadets  rated  all  of  the  packages  with  equivalent  scores  for  the  amount  of 
training  that  they  would  require.  The  tasks  that  eadets  were  assigned  did  not  take  as  mueh 
time  as  those  assigned  to  the  faeulty.  Instead  of  having  to  develop  web-enabled  eourses 
with  the  software,  they  just  had  to  navigate  courses  that  were  already  developed.  Cadets 
are  aeeustomed  to  navigating  web  sites,  so  these  tasks  were  probably  not  as  novel  to  them 
as  developing  web-enabled  eourses  were  to  the  faeulty.  Many  websites  have  unintuitive 
navigational  layouts,  and  “hunt  and  eliek”  navigation  is  frequently  used  by  people  who 
browse  the  web  (Nielsen,  1998). 

For  all  of  the  eourses,  the  eourse  doeuments  were  not  where  the  eadets  expeeted 
to  find  them,  whieh  inereased  the  amount  of  time  the  eadets  had  to  seareh  to  find  the 
appropriate  handout.  The  navigation  schemes  of  the  eourses  did  differ  with  respect  to 
layout  of  the  assignments.  Using  the  layouts  of  Courseinfo  and  WebCT,  eadets  would 
have  to  seareh  under  different  headings  to  find  all  the  handouts  and  assignments  that 
might  be  due  in  a  partieular  lesson.  For  the  Intralearn  course,  the  layout  was  ordered 


lesson  by  lesson.  Of  the  eadets  that  ehose  Intraleam  over  WebCT,  all  of  them  reported 
the  navigational  layout  as  the  defining  reason.  This  preferenee  did  not  earry  over  into  the 
Courseinfo  tests.  Even  though  some  of  the  eadets  preferred  this  navigational  layout  of  the 
Intraleam  course,  the  Intraleam  course  received  a  lower  score  for  being  intuitive  than 
both  the  courses  built  in  Courseinfo  and  WebCT. 

Of  the  cadets  who  preferred  WebCT  and  Courseinfo  to  Intraleam,  they  cited 
additional  features  available  in  both  WebCT  and  Courseinfo,  such  as  an  integrated 
calendar  and  the  ability  to  find  their  grades.  Although  cadets  took  quizzes  and  were  asked 
to  check  their  grades  using  all  three  packages,  most  of  the  cadets  were  unable  to  find  their 
grades  in  the  Intraleam  course. 

Cadets  also  rated  the  features  in  the  courses  such  as  asynchronous  and 
synchronous  discussion  groups,  integrated  calendar,  using  external  links,  taking  quizzes 
and  finding  their  grades  as  useful.  The  only  feature  that  cadets  did  not  rate  as  useful  was 
the  drop  box  for  submitting  assignments  to  their  instmctors.  Many  of  the  cadets 
commented  that  this  was  redundant  to  submitting  assignments  as  email  attachments, 
which  is  the  current  system.  Several  cadets  also  commented  that  using  a  separate  calendar 
for  courses  was  confusing,  and  that  they  would  prefer  to  have  all  of  their  assignments  and 
announcements  integrated  into  one  calendaring  system,  which  is  part  of  the  enterprise 
system  we  seek  to  develop  and  deploy  at  USAFA. 

As  with  the  faculty  tests,  features  that  the  cadets  rated  as  being  more  important 
than  the  rest  were  included  in  the  courses  that  they  favored.  However,  since  they  were  not 
taking  a  real  course,  the  grades  that  were  posted  were  few  and  of  limited  importance.  In 
addition,  instmctor  feedback  was  not  provided  with  the  grades  that  were  issued  during  the 
evaluation  process.  As  with  faculty,  finding  features  that  they  had  predetermined  as 
useful  may  have  validated  their  beliefs  that  using  these  products  to  web-enable  courses 
was  useful. 


Section  3:  Experienced  USAFA  Faculty  and  Cadets  Rate  Blackboard 
Courseinfo 

3,1  Introduction 

In  section  2  of  this  study,  both  faculty  and  cadets  evaluated  CMS  products  using  a 
discrete  set  of  criteria.  However,  performing  a  task  in  a  simulated  environment  is  not 
always  the  same  as  using  the  tool  on  a  daily  basis.  Since  the  faculty  members  and  cadets 
are  more  experienced  at  using  the  tool  after  a  semester  than  during  the  initial  choice 
phase,  they  should  provide  finer  discrimination  between  ease  of  use  and  the  usefulness  of 
the  different  features.  In  addition,  faculty  members  teaching  different  disciplines  are 
likely  to  use  the  tool  in  slightly  different  ways,  and  put  documents  in  different  areas 
providing  variety  even  within  a  standard  template.  Using  the  tools  for  an  entire  semester 
can  provide  us  data  as  to  the  frequency  of  use  of  some  of  the  features  packaged  within  the 
products,  providing  additional  data  on  the  importance  of  certain  features. 


Therefore,  the  purpose  of  this  part  of  the  study  was  to  have  faculty  members  and 
cadets  provide  final  feedback  on  using  Course  Management  System  software  and  how  it 
functioned  to  support  the  curriculum  at  USAFA  after  they  had  been  using  for  the  tool  set 
for  a  semester.  Since  all  of  the  faculty  members  preferred  Courseinfo  in  the  choice 
experiments,  this  was  the  only  CMS  analyzed  during  this  phase  of  the  study.  However, 
the  data  collected  on  features  used  and  the  faculty’s  ability  to  use  Courseinfo  to  web- 
enable  curriculum  at  USAFA,  helps  to  structure  a  list  of  requirements  for  the  role  of  CMS 
software  in  producing  an  online  educational  environment. 

3.2  Methods 

3.2.1  Instructor  evaluations 

Thirteen  Faculty  members  from  the  United  States  Air  Force  Academy  (USAFA) 
who  had  been  using  Courseinfo  to  develop  their  online  supplemental  course  materials  for 
Fall  ‘00  volunteered  to  answer  the  questionnaires  in  December  2000.  Some  of  these 
faculty  had  participated  in  the  preliminary  choice  experiments  outlined  above,  but  others 
joined  the  study  after  the  completion  of  that  phase.  As  with  the  first  phase,  these  faculty 
members  were  asked  to  rate  each  of  the  features  listed  in  Table  1  according  to  how 
important  they  were  to  producing  web-enabled  courses.  Faculty  were  then  asked  how 
often  they  used  each  of  the  features  available  in  Courseinfo  and  to  rate  them  according  to 
ease  of  use  and  usefulness  using  a  1-100  visual  analog  scale  (Miller  1999).  They  were 
also  given  a  series  of  questions  designed  to  assess  their  overall  satisfaction  using  the 
software. 

2.2.2  Cadet  evaluations 

Cadets  evaluated  the  course  in  which  they  were  enrolled.  They  were  asked  to 
report  how  often  they  used  each  of  the  features  included  in  Blackboard  Courseinfo,  and 
to  report  the  ease  of  use  and  usefulness  of  the  product  on  a  1-100  visual  analog  scale 
(Miller,  1999).  Cadets  also  answered  a  set  of  questions  designed  to  assess  their  overall 
like  or  dislike  of  Blackboard  Courseinfo  as  a  CMS,  rather  than  their  like  and/or  dislike  of 
the  content  material  of  the  course  itself.  The  responses  of  the  cadets  were  kept 
anonymous,  and  pooled  for  all  courses. 

3.2.5  Data  collection  and  analysis 

Data  for  both  the  faculty  and  cadets  were  collected  by  self-report  on  surveys  and 
entered  into  a  MS  Excel  spreadsheet.  Summary  statistics  were  calculated  using  MSExcel 
and  are  reported  as  mean  +  sd  unless  otherwise  noted.  Data  were  imported  into  SPSS  for 
further  analysis.  Eaculty  course  questionnaire  data  was  obtained  from  the  Center  for 
Educational  Excellence.  All  identifiers  were  stripped  from  the  data  so  that  the  identity  of 
the  individual  faculty  members  remained  anonymous. 

Blackboard  Courseinfo  was  housed  on  a  Sun  Ultra  Sparc  5  server  located  in  the 
Institute  for  Information  Technology  Applications  laboratory  in  Eairchild  Hall  at 
USAEA.  The  web  pages  were  accessed  through  the  academy’s  intranet.  Cadets  and 
faculty  were  instructed  to  bypass  the  proxies  for  efficiency  and  speed. 


3,3  Results 


3.3.1  Faculty  participants 

There  were  10  male  and  3  female  faeulty  members  representing  departments  of 
various  disciplines  including  Biology,  Engineering  Mechanics,  Economics  and 
Behavioral  Sciences.  Seven  of  the  participants  (54%)  had  created  home  pages  for  their 
classes  in  previous  semesters,  but  for  six  (46%)  of  the  participants  this  was  their  first 
semester  creating  web  pages  and  using  the  web  as  a  teaching  tool.  Only  two  (15%)  of  the 
faculty  maintained  additional  web  pages  not  included  in  the  Courseinfo  CMS  at  USAEA, 
and  only  23%  maintained  a  web-site  with  an  internet  service  provider  outside  USAEA. 
Seven  (54%)  of  the  faculty  participants  chose  to  use  Blackboard  this  semester  because 
they  were  part  of  the  original  choice  experiments  (Section  2  of  this  report),  whereas  six 
(46%)  of  the  faculty  chose  to  use  the  software  upon  the  recommendation  of  the  course 
director  overseeing  the  syllabus  for  their  course. 

3.3.2  Cadet  participants 

Eourteen  male  and  5  female  cadets  evaluated  the  online  resources  of  the  faculty 
members  who  were  using  Courseinfo  during  the  Eall  ’00  semester  including  courses  in 
Biology  (n=10).  Physics  (n=3).  Economics  (n=2).  Engineering  Mechanics  (n=2),  and 
Eeadership  (n=l).  Because  the  instructors  using  Courseinfo  were  teaching  courses 
targeted  to  that  level,  the  majority  (15  or  79%)  of  the  cadets  who  evaluated  the  online 
materials  were  third-class  (sophomore)  cadets,  one  first-class  (senior)  cadet  and  2  second- 
class  (junior)  cadets.  Only  8  (41.2%)  of  the  cadets  had  used  web-sites  created  by  their 
instructors  to  supplement  the  classroom  curriculum,  whereas  most  of  the  cadets  (n  =1 1, 
57.8%)  were  using  this  resource  for  the  first  time.  None  of  the  cadets  had  ever  used 
online  course  materials  authored  using  Courseinfo  CMS  software  or  other  CMS  packages 
in  previous  semesters. 

3.3.3  Rankings  of  selection  criteria  by  faculty  and  cadets 

When  experienced  faculty  and  cadets  were  asked  to  rank  features  used  to  evaluate 
the  course  management  software  packages,  both  groups  tended  to  give  most  of  the 
features  similar  scores.  The  Grand  mean  rating  given  to  the  features  by  faculty  was  4.06 
+  0.88  for  faculty  and  4.21  +  0.58  for  cadets  on  a  6  point  scale  where  1  indicated  that  the 
feature  was  of  very  little  importance,  and  6  was  very  important  or  a  “must  have”. 

Ten  features  were  rated  5.0  or  higher  and  considered  to  be  more  important  than 
the  others  to  faculty  developing  curriculum.  Most  of  these  features  were  course 
administration  features,  the  ability  to  author  content  without  the  knowledge  of  HTME, 
grading  and  quizzing  features  (Table  21).  Seven  features  received  a  score  of  3.8  or  below 
and  were  considered  less  important  including  the  ability  to  support  foreign  languages,  the 
use  of  discussion  groups,  video-teleconferencing,  virtual  field  trips  and  the  type  of 
operating  system.  The  ability  to  conform  to  external  standardization  such  as  IMS 
specifications  received  a  rating  of  2.0  and  was  designated  as  least  important. 

Table  21:  Ratings  of  course  management  software  features  that  faculty  (n  =  13)  rated 
significantly  higher  or  lower  than  the  grand  mean  for  all  features.  The  corresponding 


scores  given  by  cadets  (n=19)  are  also  shown  with  features  that  were  given  equivalent 
ratings  by  both  groups  shown  in  bold.  The  ratings  were  on  a  6  point  seale  where  1 
indieated  that  the  feature  was  of  very  little  importanee  to  web-enabled  eurrieulum,  and  6 
was  very  important  or  a  “must  have”.  The  grand  mean  seore  for  all  features  was  4.06  + 
0.88  for  faeulty  and  4.21  +  0.58  for  eadets. 


Feature 

ISBflBl 

sd 

Cadet 

sd 

more  important: 

Does  not  reouire  knowledae  of  HTML 

5.5 

0.1 

4.9 

1.2 

Class  lists  can  be  presented,  saved,  and  printed  usinq  Excel 

5.5 

0.7 

4.1 

1.5 

Student  access  to  orooress  data  available 

5.4 

1.4 

0.9 

Class  list  can  be  uploaded  as  a  file 

5.4 

0.9 

4.0 

1.5 

Table  of  contents 

5.1 

1.0 

5.0 

1.1 

Course  can  be  downloaded  for  safekeepinq 

5.1 

1.2 

4.1 

1.7 

Scores  can  be  exoorted  into  Excel 

5.1 

1.7 

5.0 

1.0 

Quizzes  automaticallv  oraded  and  entered  into  oradebook 

5.1 

1.7 

5.1 

1.1 

Online  instructor  manual 

5.0 

1.3 

4.2 

1.6 

Can  use  a  mixture  of  Question  tvoes  on  a  sinqle  quiz 

5.0 

1.5 

5.0 

0.9 

less  important: 

Can  Support  Foreiqn  Lanquaqes 

2.9 

1.8 

1.8 

Virtual  field  trios  within  discussion  oaaes 

2.7 

1.6 

3.6 

1.9 

Loqqed  synchronous  discussion 

2.5 

1.3 

1.7 

Shared  whiteboard 

2.5 

1.4 

1.8 

Adaptable  for  desktop  videoteleconferencinq 

2.4 

1.5 

3.8 

1.8 

least  important 

Searchable  asynchronous  discussion 

2.3 

1.2 

1.7 

Ooeratina  Svstem 

2.3 

5.1 

1.7 

Conforms  to  Instruction  Manaqement  System  specs 

2.0 

0.7 

3.5 

1.5 

Twelve  features  were  rated  4.8  or  higher  and  eonsidered  to  be  more  important 
than  the  others  to  eadets  for  web-enabled  curriculum.  The  features  ineluded  in  this  group 
were  the  ability  to  aeeess  grades,  several  quizzing  features  and  the  ability  to  produee  web 
resourees  without  knowing  how  to  author  HTML  (Table  22).  Thirteen  features  reeeived  a 
score  of  3.6  or  below  and  were  eonsidered  least  important  ineluding  the  ability  to 
eonform  to  external  web  development  specifieations  such  as  IMS,  email  features,  the 
ability  to  traek  how  often  and  when  a  student  aeeessed  the  online  materials,  asynehronous 
and  synehronous  discussion  pages  and  the  operating  system  used  to  run  the  software 
(Table  22 ). 


Table  22:  Ratings  of  course  management  software  features  that  eadets  (n=19)  rated 
signifieantly  higher  or  lower  than  the  grand  mean  for  all  features.  The  eorresponding 
scores  given  by  faeulty  (n=13)  are  also  shown  with  features  that  were  given  equivalent 
ratings  by  both  groups  shown  in  bold.  The  ratings  were  on  a  6  point  seale  where  1 
indicated  that  the  feature  was  of  very  little  importanee  to  web-enabled  eurriculum,  and  6 
was  very  important  or  a  “must  have”.  The  grand  mean  seore  for  all  features  was  4.06  + 
0.88  for  faeulty  and  4.21  +  0.58  for  eadets. 


Feature 

Cadet 

s.d. 

Faculty 

S.d. 

more  imoortant: 

Ability  to  add  offline  grades 

5.3 

0.8 

4.8 

1.7 

Make  auizzes  with  true-false/multiole  choice  questions 

5.3 

0.8 

4.9 

1.7 

Student  access  to  oroaress  data  available 

5.3 

0.9 

5.4 

1.4 

One  Question  at-a-time  testing  caoabilitv 

5.3 

2.5 

4.6 

1.7 

Instructor  comments  available  with  grade 

5.2 

1.0 

4.2 

1.6 

Quizzes  automaticallv  graded  and  entered  into  aradebook 

5.1 

1.1 

5.1 

1.7 

Grade  statistics  and/or  histograms 

5.0 

1.2 

4.7 

1.8 

Scores  can  be  exported  into  Excel 

5.0 

1.0 

5.1 

1.7 

Table  of  contents 

5.0 

1.1 

5.1 

1.0 

Can  use  a  mixture  of  Question  tvoes  on  a  single  guiz 

5.0 

0.9 

5.0 

1.5 

Does  not  reouire  knowledge  of  HTML 

4.9 

1.2 

5.5 

0.1 

Fill  in  the  blank 

4.8 

1.2 

4.7 

1.7 

less  important: 

Conforms  to  World  Wide  Web  Consortium  specs 

3.6 

1.5 

3.3 

0.9 

Collaborative  work  area  for  group 

3.6 

1.6 

3.4 

1.5 

Cne  to  one  course  email 

3.6 

1.6 

3.7 

1.7 

Can  track  how  often  student  accesses  pages 

3.6 

1.5 

3.9 

1.0 

Virtual  field  trios  within  disscussion  pages 

3.6 

1.9 

2.7 

1.6 

Cne  to  many  course  email 

3.6 

1.5 

4.0 

1.7 

Shared  whiteboard 

3.5 

1.8 

2.5 

1.4 

Conforms  to  IMS  specifications 

3.5 

1.5 

2.0 

0.7 

Group  presentation  area 

3.5 

1.7 

3.2 

1.6 

Individual  presentation  area/homepage 

3.4 

1.5 

3.3 

1.0 

Searchable  asynchronous  discussion 

3.3 

1.7 

2.3 

1.2 

Operating  system 

3.3 

1.7 

2.3 

5.1 

Logged  synchronous  discussion 

3.2 

1.7 

2.5 

1.3 

3.3.4  Faculty  evaluation  of  Courseinfo 

Most  of  the  faeulty  members  did  not  use  all  of  the  features  that  were  available  in 
Courseinfo.  The  most  widely  used  features  ineluded  adding  assignments  and  documents, 
adding  external  links,  creating  course  announcements  and  the  online  grade  book.  Some 
of  the  faeulty  used  the  online  quizzing,  student  tracking,  emailing  and  calendar  features, 
and  one  or  none  of  the  faeulty  used  the  discussion  boards,  chat  features  or  any  of  the 
collaborative  work  group  features  (Table  23). 


Table  23;  The  number  of  faculty  that  used  the  features  and  the  number  of  times  that  they 
used  each  feature.  N  =  the  number  of  faculty  that  used  the  feature,  and  Mean  =  the 
number  of  time  those  faculty  used  the  feature  in  a  semester. 


Frequency  usinq  the  followinq  features: 

N 

Mean 

s.d. 

Ranqe 

Student  trackinq 

7 

64.3 

58.7 

18-180 

Online  quizzes 

8 

25.9 

14.8 

9-54 

Addinq  assiqnments/documents 

11 

23.7 

10.1 

9-36 

Asynchronous  discussions  boards 

1 

36.0 

Chat  feature 

1 

36.0 

Addinq  external  links 

10 

22.5 

12.9 

9-54 

Online  qrade  book 

9 

70.0 

82.7 

18-270 

Creatinq  announcements 

10 

32.4 

20.4 

18-72 

Use  BB  to  send  email  to  the  class 

5 

27.0 

25.5 

9-72 

File  sharinq 

1 

54.0 

Drop  box 

1 

36.0 

Group  workspace 

0 

Calendar 

5 

25.2 

9.9 

18-36 

Other 

0 

Of  the  faculty  who  used  those  features,  they  rated  course  set  up,  creating 
announcements,  sending  email  to  the  class  and  grade  book  functions  as  easy  to  very  easy. 
They  rated  eight  of  the  other  features  as  somewhat  easy  to  easy,  including  adding 
assignments  and  uploading  web  pages,  composing  quizzes  and  using  the  calendar.  The 
other  features  were  rated  as  neutral,  or  were  not  used  by  the  participants  (Table  24). 


Table  24;  Experienced  faculty  (n  =  13)  mean  ease  of  use  scores  for  Courseinfo  (Cl). 
Anchors  for  this  scale  are  0  =  very  difficult,  12  =  difficult,  29  =  somewhat  difficult,  66  = 
somewhat  easy,  82  =  easy,  100  =  very  easy.  N  =  the  number  of  faculty  that  used  the 
feature,  and  Mean  =  the  average  rating  assigned  to  that  feature. 


Task 

N 

Mean 

s.d. 

Course  set  up  (uploadinq  syllabus  and  policies) 

10 

82.7 

13.0 

Addinq  assiqnments 

8 

66.3 

27.4 

Uploadinq  web  paqes 

7 

74.7 

12.4 

LInkInq  other  web  paqes 

8 

78.9 

14.0 

Composinq  quizzes 

7 

75.4 

19.4 

Administrative  functions  overall 

8 

77.1 

13.9 

Enrollinq  cadets 

8 

61.4 

29.4 

Creatinq  announcements 

11 

88.5 

11.1 

Sendinq  email  to  class 

7 

95.0 

6.5 

Settinq  up  qrade  book 

8 

87.4 

8.4 

Enterinq  qrades  into  qrade  book 

10 

73.8 

27.8 

Exportinq  qradebook  into  excell 

6 

84.8 

9.9 

Discussion  qroups 

1 

41.0 

58.0 

File  sharinq 

1 

32.5 

46.0 

Group  workspace 

0 

Calendar 

5 

77.6 

22.5 

Chat  qroup 

0 

Modifvinq  look  and  feel  of  course 

7 

75.9 

15.8 

As  with  the  original  choice  experiments,  faculty  rated  those  features  that  were 
easy  to  use  as  the  most  useful.  Of  the  faculty  who  used  those  features,  they  rated  six 
features  as  useful  to  very  useful  including  course  set  up,  adding  assignments,  uploading 
web  pages,  and  creating  announcements  (Table  25).  They  rated  five  of  the  features  as 
somewhat  useful  to  useful,  including  grade  book  functions,  enrolling  cadets, 
administrative  functions  and  composing  quizzes.  The  other  features  were  rated  as  neutral. 


Table  25:  Comparison  of  mean  usefulness  seores  given  by  faeulty  (n  =  13)  for 
Courseinfo  (Cl).  Anehors  for  this  seale  are  0  =  not  at  all  useful,  12  =  not  very  useful,  29 
=  of  limited  use,  66  =  somewhat  useful,  82  =  useful,  100  =  very  useful.  N  =  the  number 
of  faeulty  that  used  the  feature,  and  Mean  is  the  average  rating  assigned  to  that  feature. 


Task 

N 

Mean 

s.d. 

Course  set  up  (uploadinq  syllabus  and  policies) 

10 

89.6 

10.5 

Addinq  assiqnments 

7 

91.4 

18.6 

Uploadinq  web  paqes 

7 

91.4 

14.6 

LInkInq  other  web  paqes 

7 

88.9 

14.5 

Composinq  quizzes 

8 

82.5 

18.0 

Administrative  functions  overall 

9 

78.2 

18.0 

Enrollinq  cadets 

10 

77.5 

19.3 

Creatinq  announcements 

11 

84.4 

16.7 

Sendinq  email  to  class 

8 

56.0 

27.2 

Settinq  up  qrade  book 

9 

71.8 

28.7 

Enterinq  qrades  into  qrade  book 

11 

70.5 

27.4 

Exportinq  qradebook  into  excell 

7 

75.1 

18.4 

Discussion  qroups 

5 

44.0 

34.4 

File  sharinq 

5 

46.4 

24.7 

Group  workspace 

5 

46.4 

27.7 

Calendar 

9 

64.1 

26.6 

Chat  qroup 

5 

46.4 

27.7 

Modifvinq  look  and  feel  of  course 

9 

58.2 

29.0 

Nine  of  the  13  faeulty  indieated  that  it  took  them  the  same  amount  of  time  to 
prepare  for  their  eourse  using  Courseinfo  and  only  three  faeulty  members  reported  that  it 
took  them  more  time  to  prepare  for  elass  using  the  CMS.  When  asked  to  estimate  the 
payoff  in  future  class  preparations,  four  (38.5%)  of  the  faculty  indicated  that  it  would 
take  them  less  time  to  prepare  for  class  in  the  future,  now  that  they  had  some  of  the 
resources  online.  However,  ten  (61.5%)  felt  that  it  would  take  them  the  same  amount  of 
time  to  prepare  for  future  classes.  Nine  (69.2%)  of  the  faculty  indicated  they  would  use 
Courseinfo  again  for  the  spring  semester,  whereas  3  (30.8%)  indicated  that  they  would 
not  be  using  the  software  product.  This  is  in  part  due  to  the  preferences  of  the  course 
director. 

For  10  statements  designed  to  ascertain  their  overall  experience  using  Courseinfo, 
faculty  moderately  agreed  with  eight  out  of  ten  of  the  statements.  Courseinfo  had  a  good 
navigational  and  pedagogical  layout,  was  easy  to  use,  required  little  training  to  use,  as 
well  as  had  pedagogically  useful  features.  The  faculty  scored  two  of  the  10  statements  as 
neutral  including  took  a  reasonable  amount  of  time  to  accomplish  tasks  and  had  an  easy 
to  use  instructor’s  manual  (Table  26). 


Table  26:  Comparison  of  mean  overall  assessment  scores  by  faculty  (n  =  13)  for 
Courseinfo.  Anchors  for  this  scale  are  0  =  decidedly  disagree,  25  =  moderately  disagree, 
50  =  neutral,  75  =  moderately  agree,  100  =  decidedly  agree. 


This  software  package: 

N 

Mean 

s.d. 

Took  a  reasonable  amount  of  time  to  accomplish  tasks 

12 

68.8 

26.4 

Had  a  good  naviqational  layout 

13 

80.8 

11.3 

Had  a  good  pedagogical  layout 

10 

74.5 

17.6 

Had  aesthetically  pleasing  screen  displays 

13 

76.2 

22.2 

Was  easy  to  use 

13 

78.1 

15.9 

Had  an  easy  to  use  user's  manual 

7 

65.0 

21.0 

Had  instructiye  directions  on  the  screen 

12 

75.0 

25.2 

Had  pedaqoqically  useful  features 

11 

75.9 

16.9 

Was  intuitiye  in  nature 

12 

73.3 

21.0 

Needs  little  training  to  use 

13 

72.3 

21.3 

5.5.5  Cadet  Evaluation  of  Courseinfo 

Most  of  the  cadets  did  not  use  all  of  the  features  that  were  available  in 
Courseinfo.  The  most  widely  used  features  included  taking  online  quizzes,  reading 
announcements  and  checking  the  online  grade  book.  Few  of  the  cadets  accessed  the 
online  assignments  and  documents,  used  the  email  function  or  the  online  calendar 
function,  while  two  to  none  of  the  cadets  used  the  discussion  boards,  chat  features  or  any 
of  the  collaborative  work  group  features  (Table  27). 

Table  Tl\  The  number  of  cadets  that  used  the  features  and  the  number  of  times  that  they 
used  each  feature.  N  =  the  number  of  faculty  that  used  the  feature,  and  mean  indicates  the 
number  of  time  those  faculty  used  the  feature  in  a  semester. 


Frequency  using  the  foiiowing  features: 

N 

Mean 

s.d. 

range 

Online  quizzes 

15 

18.0 

6.1 

9-36 

Adding  assiqnments/documents 

4 

27.0 

9.0 

18-36 

Asynchronous  discussion  boards 

2 

22.5 

6.4 

18-27 

Chat  features 

0 

Adding  external  links 

2 

18.0 

Online  grade  book 

13 

22.8 

12.0 

9-54 

Reading  announcements 

13 

24.2 

13.4 

18-54 

Use  BB  to  send  email  to  your  instructor 

4 

27.0 

10.3 

18-36 

File  sharing 

0 

Drop  box 

1 

36.0 

Group  workspace 

0 

Calendar 

5 

28.8 

16.1 

18-54 

Of  the  cadets  who  used  those  features,  they  rated  eight  of  the  twelve  features  as 
easy  to  very  easy  to  use  including  locating  announcements  and  course  documents, 
sending  email  to  the  class  and  checking  their  grade.  They  rated  the  other  features  as 


somewhat  easy  to  easy,  ineluding  turning  in  an  assignment  using  the  drop  box,  using  the 
discussion  boards,  and  using  the  calendar  (Table  28). 


Table  28;  Experienced  cadet  (n  =  19)  mean  ease  of  use  scores  for  Courseinfo  (Cl). 
Anchors  for  this  scale  are  0  =  very  difficult,  12  =  difficult,  29  =  somewhat  difficult,  66  = 
somewhat  easy,  82  =  easy,  100  =  very  easy.  N  =  the  number  of  cadets  who  rated  the 
feature,  and  Mean  =  the  average  rating  assigned  to  that  feature. 


Task 

N 

Mean 

s.d. 

Locatina  the  syllabus 

16 

80.9 

20.7 

Locatinq  the  announcements 

18 

86.8 

19.5 

Locatinq  Instructor  Information 

18 

85.4 

19.2 

Locatinq  the  course  policies 

17 

83.6 

19.2 

Locatinq  the  assiqnments 

19 

88.9 

14.8 

Turninq  In  an  assiqnment  usinq  the  drop  box 

8 

62.4 

37.9 

Locatinq  the  external  links 

14 

83.4 

17.5 

Usinq  the  discussion  boards 

7 

65.6 

35.7 

Usinq  the  calendar  function 

10 

69.9 

35.4 

Takinq  quizzes 

16 

91.1 

14.8 

Checkinq  your  qrade 

16 

87.4 

15.0 

Emallinq  the  Instructor 

8 

81.3 

18.4 

The  only  Courseinfo  feature  that  the  cadets  rated  as  useful  to  very  useful  was 
checking  their  grade  (Table  29).  They  rated  seven  of  the  features  as  somewhat  useful  to 
useful  including  locating  the  syllabus,  announcements  and  assignments,  taking  quizzes 
and  emailing  the  instructor.  The  other  features  were  rated  as  neutral. 

Table  29:  Comparison  of  mean  usefulness  scores  given  by  cadets  (n  =  19)  for  Courseinfo 
(Cl).  Anchors  for  this  scale  are  0  =  not  at  all  useful,  12  =  not  very  useful,  29  =  of  limited 
use,  66  =  somewhat  useful,  82  =  useful,  100  =  very  useful.  N  =  the  number  of  cadets  who 
rated  the  feature,  and  Mean  is  the  average  rating  assigned  to  that  feature. 


Task 

N 

Mean 

s.d. 

Locatinq  the  syllabus 

15 

63.7 

32.1 

Location  the  announcements 

16 

70.6 

28.6 

Locatinq  Instructor  Information 

16 

68.9 

28.1 

Locatinq  the  course  policies 

15 

58.0 

34.4 

Locatinq  the  assiqnments 

17 

77.8 

28.1 

Turninq  In  an  assiqnment  usinq  the  drop  box 

9 

39.1 

47.4 

Locatinq  the  external  links 

10 

63.8 

34.6 

Usinq  the  discussion  boards 

9 

40.2 

39.9 

Usinq  the  calendar  function 

10 

56.6 

41.7 

Takinq  quizzes 

15 

76.7 

33.8 

Checkinq  your  qrade 

15 

85.4 

25.6 

Emallinq  the  instructor 

9 

62.9 

38.7 

Thirteen  of  the  19  cadets  indicated  that  it  took  them  the  same  amount  of  time  to  prepare 
for  their  course  when  the  course  was  supplemented  with  materials  prepared  using 


Courseinfo  when  compared  to  other  courses  taught  with  non-CourseInfo  web  sites  and 
four  cadets  indicated  that  it  took  them  up  to  four  hours  a  week  less.  Only  2  cadets 
reported  spending  up  to  1.5  hours  more  each  week  in  class  preparation.  Fourteen  (73.6%) 
of  the  cadets  indicated  they  would  like  to  use  Courseinfo  again  during  the  spring 
semester,  whereas  five  (26.4%)  indicated  that  they  would  rather  not  use  web  sites 
produced  by  the  software  product. 

For  10  statements  designed  to  ascertain  their  overall  experience  using  Courseinfo, 
cadets  moderately  agreed  with  eight  out  of  ten  of  the  statements.  Courseinfo  had  a  good 
navigational  and  logical  layout,  was  easy  to  use,  required  little  training  to  use,  as  well  as 
had  useful  features.  The  cadets  scored  two  of  the  10  statements  as  neutral  including  took 
a  reasonable  amount  of  time  to  accomplish  tasks  and  had  an  easy  to  use  manual  (Table 
30). 


This  software  package: 

N 

Mean 

s.d. 

Took  a  reasonable  amount  of  time  to  accomplish  tasks 

18 

66.1 

30.3 

Had  a  good  naviqational  layout 

19 

75.7 

24.6 

Had  a  logical  layout 

19 

72.3 

24.5 

Had  aesthetically  pleasing  screen  displays 

19 

83.8 

13.4 

Was  easy  to  use 

19 

80.5 

23.4 

Had  an  easy  to  use  user's  manual 

16 

54.6 

34.7 

Had  instructiye  directions  on  the  screen 

18 

73.3 

25.5 

Had  useful  features 

19 

73.1 

24.8 

Was  intuitiye  in  nature 

19 

72.1 

26.4 

Needs  little  training  to  use 

19 

77.3 

29.0 

Table  30:  Comparison  of  mean  overall  assessment  scores  by  cadets  (n  =  19)  for 
Courseinfo.  Anchors  for  this  scale  are  0  =  decidedly  disagree,  25  =  moderately  disagree, 
50  =  neutral,  75  =  moderately  agree,  100  =  decidedly  agree. 


3.3. 6  End  of  course  assessment  data 

Faculty  members  who  used  Courseinfo  received  slightly  higher  course 
evaluations  than  other  faculty  members  during  the  Fall  ’00  semester  (Figure  1)  for  ten 
questions  chosen  to  assess  teaching  performance.  However,  those  thirteen  faculty 
members  also  received  slightly  higher  course  evaluations  than  the  rest  of  the  faculty  in 
the  semester  before  they  began  to  use  Courseinfo.  In  one  of  the  core  courses,  half  of  the 
faculty  used  Courseinfo  as  a  supplemental  teaching  tool  and  the  other  faculty  did  not  use 
web  pages  to  supplement  their  teaching.  Faculty  members  who  were  using  Courseinfo 
received  higher  end  of  course  assessments  than  faculty  who  were  not  for  that  course 
(Figure  2). 


Figure  1:  The  end  of  course  assessment  data  for  the  thirteen  Courseinfo  users  during  the 
semester  they  were  using  the  product  compared  with  all  faculty  during  the  same  semester 
and  the  semester  before  they  were  using  the  product.  The  ratings  were  based  on  a  1  to  6 
scale  with  1  equal  to  very  poor  and  6  equal  to  excellent.  None  of  the  scores  were 
statistically  significant  from  the  other  scores  at  the  0.05  level. 
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Question; 

I .  Instructor’s  ability  to  stimulate  my  interest  was: 

3.  Instructor’s  ability  to  provide  clear,  well-organized  instruction  was; 

7.  Instructor’s  knowledge  of  course  material  was: 

8.  Asa  military  role  model  or  civilian  professional  role  model,  my  instructor  was: 

I I .  Availability  of  extra  help  when  needed  was: 

13.  Course  organization  was: 

14.  Clarity  of  course  objectives  and  requirements  was: 

19.  Quality  and  usefulness  of  course  text(s)  were; 

20.  The  course  as  a  whole  was: 

23.  The  instructor’s  effectiveness  in  facilitating  my  learning  in  the  course  was: 

Figure  2:  Comparison  of  end  of  course  assessment  data  for  4  core  course  instructors 
who  used  Blackboard  Courseinfo  compared  with  4  instructors  teaching  the  same  course 
who  did  not  use  the  software  or  other  web  site  supplementation.  The  ratings  were  based 
on  a  1  to  6  scale  with  1  equal  to  very  poor  and  6  equal  to  excellent.  The  differences 
between  the  scores  for  questions  8,  14,  and  19  were  statistically  significant  at  the  0.05 
level. 
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Question; 

I.  Instructor’s  ability  to  stimulate  my  interest  was: 

3.  Instructor’s  ability  to  provide  clear,  well-organized  instruction  was: 

7.  Instructor’s  knowledge  of  course  material  was: 

8.  As  a  military  role  model  or  civilian  professional  role  model,  my  instructor  was: 

I I .  Availability  of  extra  help  when  needed  was: 

13.  Course  organization  was: 

14.  Clarity  of  course  objectives  and  requirements  was: 

19.  Quality  and  usefulness  of  course  text(s)  were; 

20.  The  course  as  a  whole  was: 

23.  The  instructor’s  effectiveness  in  facilitating  my  learning  in  the  course  was: 

3,4  Discussion 

3.4.1  Faculty  and  cadet  rankings  of  selection  criteria 

When  participants  who  had  been  using  Courseinfo  for  one  semester  were  asked  to 
determine  which  features  were  important  and  should  be  included  in  software  to  facilitate 


the  development  and  use  of  web-enabled  eourses,  both  faeulty  members  and  eadets 
followed  the  trend  of  partieipants  from  the  ehoiee  experiments  and  foeused  on  their  own 
requirements.  Cadets  foeused  on  the  types  of  quiz  questions  that  would  appear  on  exams 
and  the  ability  to  eheek  their  seores  online,  and  the  faeulty  eontinued  to  foeus  on 
administrative  proeedures  and  quiz  eonstruetion.  However,  there  was  more  agreement 
between  these  two  groups  than  from  the  previous  experiments.  Both  groups  felt  that 
student  aeeess  to  progress  data,  automatieally  graded  quizzes,  using  a  mixture  of  question 
types  on  a  single  quiz,  table  of  eontents  and  the  ability  to  ereate  pages  without  the 
knowledge  of  HTML  were  more  important  than  other  features  required  of  a  CMS 
produet. 

The  faeulty  members  and  the  eadets  eontinued  to  rate  the  diseussion  groups, 
shared  whiteboards,  virtual  field  trips  and  the  type  of  operating  system  on  whieh  the 
CMS  will  operate  as  features  that  are  not  neeessary,  or  less  important  than  most  of  the 
other  features.  In  addition,  faeulty  eontinued  to  rate  the  support  of  foreign  languages  as 
less  important,  although  there  were  no  foreign  language  instruetors  represented  in  this 
population.  The  importanee  of  a  CMS  to  support  foreign  language  instruetion  has  been 
indieated  in  a  separate  eommunieation  as  a  requirement  by  the  foreign  language 
department. 

Previously,  eadets  rated  all  student  traeking  features,  ineluding  how  often  a 
student  aeeesses  pages,  time  spent  on  eaeh  page,  and  when  a  student  aeeesses  pages  as 
less  important  than  other  features.  However,  eadets  who  were  enrolled  in  eourses  using 
Courseinfo  did  not  list  these  features  as  less  important.  Cadets  who  had  not  used  the 
system  prior  to  rating  the  features  may  have  been  eoneerned  how  faeulty  would  use  this 
information,  but  eadets  enrolled  in  eourses  using  Courseinfo  did  not  seem  to  objeet  to  the 
way  this  information  was  used  by  those  faeulty  members  who  ehose  to  aeeess  student 
traeking  features. 

3.4.2.  Experienced  faculty  ratings  of  Blackboard  Courseinfo 

Most  of  the  faeulty  used  Courseinfo  to  post  doeuments  and  assignments,  to  send 
their  eadets  announeements,  add  external  links,  to  report  seores  and  to  administer  online 
quizzes.  Of  these  eommonly  used  tasks,  the  faeulty  rated  most  of  them  as  somewhat  easy 
to  easy  to  use,  as  well  as  somewhat  to  very  useful.  When  asked  what  they  most  liked 
about  using  Courseinfo,  most  of  the  faeulty  reported  the  use  of  automatieally  graded 
quizzes,  eadet  aeeess  to  grade  book,  and  a  eentral  plaee  to  post  announeements  and 
assignments.  Although  faeulty  used  the  online  grade  book,  many  had  problems  using  it  as 
designed  and  found  it  more  eumbersome  to  enter  grades  into  the  program  than  into  a 
spreadsheet  sueh  as  MSExeel. 

Very  few  of  the  faeulty  used  the  eollaborative  features  of  Courseinfo  ineluding 
the  diseussion  boards,  ehat  feature,  fide  sharing,  drop  box  or  group  workspaee.  Of  those 
who  rated  those  features,  they  were  rated  as  neutral  with  respeet  to  ease  of  use  and 
usefulness.  When  asked  what  feature  they  would  delete,  ten  (76.9%)  indieated  that  they 
would  not  delete  any  of  the  eomponents  and  only  three  (23.1%)  said  they  would  delete 
any  of  the  features,  naming  the  ehat  and  diseussion  boards  speoifieally.  As  stated  in  an 


earlier  seetion  of  this  report,  that  may  be  influeneed  by  our  use  of  the  produet  to 
supplement  traditional  elassroom  edueation,  rather  than  to  deliver  distanee  or  online 
eourses. 

Most  faeulty  members  were  satisfied  with  the  produet  overall  and  moderately 
agreed  that  it  had  a  good  pedagogieal  layout,  pedagogieally  useful  features,  and  that  it  did 
not  take  them  more  time  to  prepare  for  elass  than  to  prepare  for  a  similar  eourse  without 
Courseinfo  web-enabled  eurrieulum  materials.  Many  faeulty  members  indieated  that  it 
laeked  some  key  administrative  features,  for  example  there  was  no  way  to  divide  grade 
books,  announeements  or  doeuments  into  multiple  seetions  of  the  same  eourse  without 
duplieating  the  entire  eourse.  Similarly,  it  was  not  possible  to  share  ealendars,  quiz 
questions,  handouts,  or  eontent  among  instruetors  teaehing  the  same  eourse.  Therefore, 
this  software  seems  best  suited  for  providing  web-enabled  instruetional  support  to  a 
faeulty  member  who  is  teaehing  one  seetion  of  a  single  eourse,  and  who  does  not  share 
large  amounts  of  eontent  with  instruetors  teaehing  other  seetions. 

3.4.3  Experienced  cadet  ratings  of  Blackboard  Courseinfo 

Most  of  the  eadets  used  Courseinfo  to  take  quizzes,  read  announeements  and 
aeeess  their  grades.  Of  these  eommonly  used  tasks,  eadets  rated  most  of  them  as  easy  to 
very  easy  to  use,  as  well  as  somewhat  useful  to  useful.  When  asked  what  they  liked  most 
about  using  Courseinfo,  most  of  the  eadets  reported  the  use  of  automatieally  graded 
quizzes  with  instant  feedbaek,  aeeess  to  grade  book,  a  eentral  loeation  for  all  materials 
and  eourse  information,  and  that  misplaeed  handouts  or  ones  not  pieked  up  in  elass  eould 
be  printed  out  off  the  web  site.  One  eadet  also  eommented  that  using  Courseinfo  also 
wasted  less  paper. 

Very  few  or  none  of  the  eadets  used  the  interaetive  features  of  Courseinfo 
ineluding  the  diseussion  boards,  ehat  feature,  fide  sharing,  drop  box  or  group  workspaee, 
presumably  beeause  they  were  not  assigned  by  their  instruetor.  Of  those  who  rated  those 
features,  they  rated  them  as  somewhat  easy  to  use  and  neutral  with  respeet  to  usefulness. 
When  asked  what  feature  they  would  delete,  only  three  (15.8%)  indicated  that  they  would 
delete  anything;  two  of  these  cadets  would  delete  the  chat  and  discussion  boards,  and 
another  cadet  commented  that  they  would  delete  the  campus  and  community  center  pages 
(these  sections  were  not  part  of  the  this  test,  and  were  therefore  not  populated  with 
events). 

While  most  cadets  were  satisfied  with  the  product  overall,  and  moderately  agreed 
that  it  was  easy  to  use,  had  a  good  logical  layout,  useful  features,  and  that  it  did  not  take 
them  more  time  to  prepare  for  class  than  not  using  the  web-based  support  materials, 
many  cadets  expressed  disappointment  that  the  faculty  members  were  not  using  the 
software  to  its  fullest  extent.  Some  indicated  that  they  would  like  to  see  more  than  just 
class  notes  and  announcements  posted.  Cadets  expressed  interest  in  online  quizzes  and 
answers,  and  others  expressed  interest  in  the  chat  features.  Although  some  of  their 
requests  may  have  been  unrealistic  (“I  really  don’t  like  homework  so  if  Blackboard  could 
stop  it,  that  would  be  nice”),  this  indicates  that  cadets  are  comfortable  with  web-enabled 


curriculum  support  materials,  and  would  be  not  be  adverse  to  future  developments  in  this 
area. 


Even  though  Courseinfo  was  housed  on  an  intranet  server  inside  the  USAFA 
firewall,  the  largest  problem  eited  by  eadets  was  the  reliability  of  the  eonneetion  to  the 
web-server.  Cadets  reported  error  messages  that  they  misinterpreted  to  mean  the 
Courseinfo  server  was  down  over  the  duration  of  the  entire  semester.  However,  the  server 
was  down  only  one  time  when  elasses  were  in  session  during  the  duration  of  the  study 
(April  to  Deeember,  2000).  Error  messages  were  more  than  likely  eaused  by  other 
eonneetions  within  USAFAnet  outside  the  researeh  lab  housing  the  Courseinfo  server. 
Cadets,  like  most  eomputer  users,  laek  the  time,  knowledge  and  adequate  system 
information  to  properly  troubleshoot  the  problem,  and  therefore  any  network  problem 
equally  impedes  their  ability  to  aeeess  the  information  and  use  the  system.  Before  any 
CMS  ean  be  deployed  at  the  enterprise  level,  this  problem  will  have  to  be  examined 
further  and  even  the  pereeption  of  interrupted  server  aeeess  must  be  resolved. 

3.4.4  End  of  course  critique  evaluations  for  instructors  using  Blackboard  Courseinfo 

The  faeulty  members  who  ehose  to  partieipate  in  this  study  seored  higher  than 
most  instruetors  on  end  of  eourse  assessments  before  they  volunteered  to  use  Courseinfo, 
as  well  as  after  they  began  using  the  produet.  Their  seores  were  higher  for  questions  that 
might  depend  on  eurriculum  delivery  method,  sueh  as  those  pertaining  to  eourse 
organization  and  the  ability  to  provide  elear,  well-organized  instruetion  as  well  as 
questions  that  should  be  independent  of  using  Courseinfo,  sueh  as  the  ability  of  the 
instructor  to  be  a  role  model  and  knowledge  of  eourse  material.  This  indieates  that  the 
software  was  evaluated  by  some  of  the  “better”  teaehing  faeulty,  and  that  using  the 
software  did  not  prevent  them  from  providing  quality  instruetion  to  the  eadets.  In 
addition,  most  of  the  faculty  participants  had  higher  assessment  seores  after  they  started 
using  Courseinfo  than  before,  but  it  is  unelear  whether  that  was  due  to  the  structure 
provided  to  the  eourse  from  the  software  product,  the  web-enabling  of  the  eourse  itself, 
the  expertise  that  eomes  with  teaehing  a  course  multiple  times,  the  inereased  attention 
and  enthusiasm  paid  to  the  course  by  the  instruetors  as  they  tried  a  new  teaehing  method, 
or  some  other  variable. 

A  direet  eomparison  of  four  instruetors  who  used  Courseinfo  to  provide  web- 
enabled  eontent  to  four  instructors  who  taught  sections  of  the  same  eourse,  but  did  not 
provide  any  supplemental  web-based  materials,  showed  that  the  instruetors  using 
Courseinfo  had  higher  end  of  semester  eourse  eritique  seores  than  the  other  group.  The 
higher  seores  ineluded  elarity  of  eourse  objeetives  and  quality  and  usefulness  of  eourse 
texts.  This  may  be  due  to  supplementation  of  the  course  text  materials  with  online 
readings  and  examples.  These  instructors  also  received  higher  ratings  for  being  a  role 
model,  whieh  should  not  be  dependent  on  using  web-enabled  eurrieulum.  It  is  diffieult  to 
tease  apart  the  various  faetors  that  eontribute  to  higher  ratings,  and  it  may  be  that  the 
eourse  eritiques  for  these  seetions  have  nothing  to  do  with  using  the  technology  used  to 
enhanee  the  eourses.  However,  it  does  appear  from  these  results  that  this  technology  does 
not  detraet  from  an  instruetor’s  ability  to  provide  quality  instruetion. 


Section  4:  Overall  Discussion  and  Conclusions 


Faculty  members  and  eadets  found  that  course  management  software  (CMS) 
provided  a  satisfactory  mechanism  for  web-enabled  curriculum  delivery  to  supplement 
traditional  classroom  instruction  at  USAFA.  Faculty  were  able  to  learn  one  software 
package  for  web-based  eurriculum,  assessments,  collaborative  features  and  for 
communicating  with  their  students.  They  did  not  have  to  learn  how  to  use  HTML,  and  as 
illustrated  by  the  eourse  eritique  assessment  data,  the  technology  did  not  overshadow  the 
teaehing  and  learning  of  course  content. 

If  we  had  only  focused  on  checklists  of  what  the  software  produets  could 
do,  then  Blackboard  Courseinfo  and  WebCT  would  have  been  considered  equal,  and 
Intralearn  would  have  been  a  distant  third.  However,  Blackboard  Courseinfo  received 
higher  scores  than  the  other  products  when  ratings  were  based  on  user  analysis  of  ease  of 
use  and  usefulness. 

Many  of  the  features  found  in  these  products  such  as  discussion  groups,  student 
web  pages  and  collaborative  work  areas  that  eontributed  to  their  high  number  of  features 
were  not  widely  used,  nor  deemed  important  by  both  faculty  and  cadets.  Although  some 
faculty  may  eventually  use  these  features  as  they  beeome  more  familiar  with  the  software 
and  pedagogy,  at  this  time  a  CMS  with  an  easy  to  use  interface  that  eontains  a  grade 
book,  automated  quizzes  and  a  place  to  put  announcements  and  course  doeuments  should 
be  preferred  to  one  that  contains  many  collaboration  features,  yet  has  a  difficult 
navigational  interface  or  hard  to  use  development  tools.  Once  the  faeulty  has 
incorporated  the  teehnology  into  the  eurriculum,  there  is  a  higher  probability  that  the 
cadets  would  then  find  it  useful. 

As  predicted,  the  faculty  and  cadets  had  different  requirements  of  the  CMS 
produet  for  developing  and  accessing  resources,  respectively.  Therefore,  if  only  the 
faeulty ’s  requirements  are  considered,  or  if  the  product  is  not  evaluated  by  all  of  the  user 
groups,  the  probability  of  purchasing  a  sub-optimal  product  for  a  majority  of  the  users 
increases.  Although  the  requirements  as  stated  by  the  faculty  and  cadets  who  had  been 
using  a  CMS  for  an  entire  semester  were  more  similar  than  those  gathered  from  the  naive 
users,  often  enterprise  CMS  purchase  decisions  are  made  prior  to  the  purchasing  of  any 
system. 


To  fully  take  advantage  of  the  benefits  of  a  CMS,  there  must  be  buy  in  from  as 
much  of  the  institution  as  possible,  and  course  information  should  be  integrated  with 
other  institutional  databases  into  one  system  using  an  academic  portal  or  other  similar 
interface.  Since  one  advantage  of  this  type  of  software  is  that  the  students  only  have  to 
learn  one  navigational  interface  and  information  management  system,  those  benefits  may 
be  deereased  when  more  than  one  system  with  different  navigational  and  features  is 
deployed.  However,  to  effectively  deploy  only  one  system  here  at  USAFA,  we  will  need 
to  be  able  to  effeetively  share  content  among  instructors  teaching  the  same  course,  and 
ereate  grade  books,  announcements  and  other  materials  for  speeifie  seetions  of  multi¬ 
section  courses  without  duplieation  of  the  entire  course. 


Section  5:  Phase  Two  of  the  CPEG  Portal  Project 


The  results  obtained  in  this  study  will  help  us  determine  the  requirements  for 
proeuring  eourse  management  system  software  for  use  here  at  USAFA.  Giving  faeulty  an 
effieient  means  of  entering  their  homework  assignments  into  a  database  system  is  the  first 
and  probably  most  important  step  to  aehieving  the  vision  of  an  integrated  portal  system 
that  ineludes  the  resourees  eadets  need  most;  aceess  to  homework  and  reading 
assignments,  eourse  materials  and  the  ability  to  monitor  their  own  progress  through 
eheeking  their  grades. 

Following  along  the  results  of  this  researeh,  the  portal  system  that  is  being 
developed  will  ineorporate  a  user-based  design  and  analysis  taking  into  aeeount  the  needs 
of  all  users.  To  maximize  efficieney,  we  will  use  a  database  model  to  faeilitate  the 
sharing  and  reusability  of  content.  The  system  will  be  organized  around  a  calendar  that 
will  tap  into  many  scheduling  databases  currently  housed  independently  here  at  USAFA 
as  well  as  the  course  management  system  (Figure  3).  To  maximize  ease  of  use,  the 
system  will  have  a  single  sign-on  and  use  the  inverse  click  rule  to  insure  that  data  most 
important  to  the  user  will  take  the  fewest  number  of  mouse  clicks  to  access. 

The  portal  design  will  break  up  the  desktop  in  to  a  series  of  seamless  blocks  or 
portlets.  This  will  allow  the  information  displayed  on  the  desktop  to  be  personalized  to 
the  user  and  optional  blocks  such  as  personal  links  and  task  lists  can  be  customized  by  the 
user.  Although  this  project  is  still  under  development,  we  anticipate  a  prototype  to  be 
developed  during  Summer  2001  and  undergo  usability  testing  during  the  2001-02 
academic  year. 


Figure  3:  Prototype  design  for  the  USAFA  portal  or  Cadet  Personalized  Educational 
Gateway  system. 


Section  6:  Acknowledgements 

I  wish  to  thank  the  cadets  and  faculty  who  gave  up  their  time  to  evaluate  the 
software  and  to  fdl  out  numerous  survey  forms,  especially  the  13  instructors  who  used 
Blackboard  Courseinfo  during  the  fall  semester  and  provided  very  thoughtful  feedback  at 
many  stages  during  the  process.  I  also  wish  to  thank  J.  Miller  for  help  with  the 
experimental  and  survey  design,  D.  Chuba  and  S.  Collins  for  their  help  in  data  collection, 
entry  and  analysis,  and  D.  Portillo  for  laboratory  support.  Many  people  provided 
comments  on  an  earlier  draft  of  this  manuscript,  including  R.  Enger  and  J.  McCarthy. 
Their  comments  were  appreciated  and  significantly  improved  the  final  product. 


7.  Literature  Cited 


Anderberg,  A.  2001.  History  of  the  internet  and  the  web.  Retrieved  April  9,  2001  from 
the  WWW;  http ://www. anderbergfamilv.net/ ant/history/. 

Brown,  D.  2000.  Interactive  Learning  in  Vignettes  from  America ’s  Most  Wired 
Campuses.  Anker  Publishing  Company,  Boston.  282  pp. 

Campbell,  J.  2000.  Poetry  in  motion;  The  Rilke  project  in  Vignettes  from  America ’s  Most 
Wired  Campuses.  Anker  Publishing  Company,  Boston.  267-269. 

Gray,  S.  1998.  Web-based  Instructional  Tools.  Syllabus  Magazine;  Volume  12  (2). 

Halloran,  1999a.  A  primer  to  Interactive  WWW-based  tutorials,  in  Pikes  Peak 
Educational  Innovations  and  Research  Symposium  Procedings.  Human- 
Environmental  Research  Center  1999-01.  United  States  Air  Eorce  Academy. 

Halloran,  1999b.  Cadet  Personalized  Educational  Gateway.  Design  and  Evaluation  of  the 
User  Interface.  Institute  for  Information  Technology  Applications  Research  Paper 
99-1.  United  States  Air  Eorce  Academy. 

Hazari,  S.l.  2001.  Evaluation  and  selection  of  web  course  management  tools.  Retrieved 
January  25,  2001  from  the  World  Wide  Web;  http;//sunil.umd.edu/webct. 

InfoWorld.  1998.  The  virtual  classroom;  web-based  training  solutions.  InfoWorld 
November  23,  1998. 

Kristapiazzi,  G.  1999.  Compare  web  tools  for  course  authoring.  Technical  report 

produced  for  Daytona  Beach  Community  College.  Retrieved  August  25,  1999  from 
the  World  Wide  Web;  http://www.geocities.cotn/Eureka/Gold/6Q12/compare  web  tools.html. 

Landon,  B.  2000.  On  line  educational  delivery  applications;  A  web  tool  for  comparative 
analysis.  Retrieved  December  20,  2000  from  the  World  Wide  Web; 
http;//www.ctt.bc.ca/landonline/. 

Marshall  University.  2000.  Comparison  of  online  course  delivery  software  products. 
Retrieved  August  25,  1999  from  the  World  Wide  Web; 
http;//multimedia.marshall.edu/cit/webct/compare/comparison.html. 

Miller,  J.  1999.  Creation  of  visual  analog  scales  based  upon  the  Army  Questionnaire 

Manual.  Human-Environmental  Research  Center  Technical  Report  1999-03.  United 
States  Air  Eorce  Academy. 

Nielsen,  J.  1998.  Testing  whether  web  page  templates  are  helpful.  Retrieved  August  27, 
1999  from  the  World  Wide  Web;  http;//www.useit.com/alertbox/9805 17.html. 


Nielsen,  J.  2000.  Why  you  only  need  to  test  with  5  users.  Retrieved  March  19,  2000  from 
the  World  Wide  Web:  http://www.useit.eom/alertbox/20000319.html. 

Novack,  G,  E.  Patterson,  A.  Gavrin  and  W.  Christian.  1999.  Just-In-Time  Teaching, 
Blending  Active  Learning  with  Web  Technology.  Prentice  Hall.  188  pp. 

Poe,  E.  2000.  Industrial  Technology,  Internet  research  tool,  in  Vignettes  from  America ’s 
Most  Wired  Campuses.  Anker  Publishing  Company,  Boston,  pp  91-92. 

Stewart,  W.  2000.  How  the  web  was  invented.  Retrieved  January  5,  2001  from  the  World 
Wide  Web:  http://hvinginternet.com/. 

Sup  inski,  S.,  Sutherland,  R.  and  S.  Valentine.  1999.  Russian  language  development  and 
maintenance  at  a  distance:  Methodology  and  technology.  Institute  for  Information 
Technology  Applications  Research  Publication  2.  United  States  Air  Eorce 
Academy. 

Taylor,  M.  2000.  Cyberscapes,  in  Vignettes  from  America’s  Most  Wired  Campuses. 
Anker  Publishing  Company,  Boston.  273-274. 


Appendix  1:  Ratings  of  the  importance  of  different  course  management  system  features 
by  faculty  and  cadets  before  and  after  they  used  Course  Management  Software.  The 
ratings  were  on  a  6  point  scale  where  1  indicated  that  the  feature  was  of  very  little 
importance  to  web-enabled  curriculum,  and  6  was  very  important  or  a  “must  have”. 


General  Features _ 

Static  toolbar _ 

Table  of  Contents 

Customized  "look  and  feel"  (colors,  icons,  logos,  etc) _ 

Automated  glossary  tool _ 

Automated  link  to  course  material  content 
Automated  indexing  tool 

Search  tool  for  course  material _ 

Student  can  make  private  annotations  of  course  material _ 

Integrated  Calendar  tool 

Can  support  foreign  languages _ 

Instructor  can  define  groups  of  students _ 

-  Can  assign  specific  material  to  individual  or  group  of  students 

-  Collaborative  work  area  for  group _ 

-  Group  presentation  area _ 

Individual  presentation  area/homepage _ 

Student  fde  upload  capability/instmctor  comments 
Conforms  to  Instructional  Management  System  (IMS) 
specifications 

Conforms  to  World  Wide  Web  Consortium  (W3C)  specifications 
Other: 

Authoring  features _ 

Does  not  require  knowledge  of  HTML _ 

Allows  Java  applets _ 

Requires  plug-ins _ 

Has  drag  and  drop  authoring  features _ 

Allows  Macromedia  files _ 

Allows  Frontpage  files 

Allows  links  to  custom  developed  pages _ 

Allows  custom  graphics 


Faculty 

Cadets 

4.0 

5.1 

5.1 

5.3 

4.4 

4.2 

4.2 

4.1 

5.3 

5.2 

4.1 

4.2 

5.1 

5.4 

4.4 

3.7 

4.2 

3.8 

3.0 

3.0 

4.6 

3.8 

4.8 

4.2 

4.5 

4.1 

4.5 

4.0 

4.3 

3.9 

4.6 

4.7 

3.4 

3.1 

4.1 

3.7 

4.6 

4.5 

4.3 

4.5 

3.5 

3.4 

5.2 

4.6 

4.3 

4.1 

4.7 

4.3 

5.3 

4.6 

5.3 

4.4 

Course  Management  Features 

Class  list  can  be  entered  one  student  at  a  time,  or  uploaded  as  a  file 

5.3 

4.6 

4.4 

4.3 

Class  list  can  be  uploaded  as  a  fde 

5.2 

4.7 

5.4 

4.0 

Class  lists  can  be  presented,  saved,  and  printed  using  Excel 

5.3 

4.7 

5.5 

4.1 

Online  student  manual 

4.8 

5.4 

4.9 

4.4 

Online  instructor  manual 

4.9 

4.8 

5.0 

4.2 

Course  can  be  downloaded  for  safekeeping 

5.4 

4.9 

5.1 

4.1 

Courses  can  easily  be  moved  from  one  server  to  another 

4.9 

4.6 

4.4 

4.2 

Directory  upload  capability  from  desktop 

5.2 

4.3 

4.4 

3.9 

Student  tracking  features 

5.0 

3.0 

4.3 

4.0 

-  can  track  how  often  student  accesses  pages 

4.7 

3.0 

3.9 

3.6 

-  can  track  when  student  accesses  pages 

4.6 

2.9 

3.7 

3.7 

-  can  track  time  spent  on  each  page 

4.4 

3.0 

3.7 

3.7 

Other: 

Communication  Features 

One  to  one  course  email 

5.0 

4.7 

3.7 

3.6 

One  to  many  course  email 

5.1 

4.9 

4.0 

3.6 

Searchable  asynchronous  discussion 

4.1 

3.5 

2.3 

3.3 

Logged  synchronous  discussion 

3.8 

3.3 

2.5 

3.2 

Vitual  field  trips  within  discussion  pages 

3.8 

3.8 

2.7 

3.6 

Shared  whiteboard 

4.1 

4.0 

2.5 

3.5 

Adaptable  for  desktop  videoteleconferencing 

3.8 

3.7 

2.4 

3.8 

Other: 

Grading  Features 

Student  access  to  progress  data  available 

5.0 

5.6 

4.4 

5.3 

Ability  to  add  offline  grades 

5.0 

5.0 

4.8 

5.3 

Grade  statistics  and/or  histograms 

4.9 

4.7 

4.7 

5.0 

Instructor  comments  available  with  grade 

5.0 

5.1 

4.2 

5.2 

Scores  can  be  emailed  to  instructor 

5.0 

4.6 

4.6 

4.7 

Scores  can  be  stored  on  server 

4.8 

4.8 

4.6 

4.8 

Scores  can  be  exported  into  Excel 

5.2 

4.7 

5.1 

5.0 

Other: 

Quizzing  features 

Quizzes  automatically  graded  and  entered  into  gradebook 

5.3 

4.9 

5.1 

5.1 

Allows  for  the  following  automatically  graded  question  types: 

-  Tme  -False/Multiple  choice 

5.2 

5.1 

4.9 

5.3 

-  Fill  in  the  blank 

5.1 

4.0 

4.7 

4.8 

-  List  matching 

5.2 

5.0 

4.7 

5.1 

-  Essay  questions 

4.8 

3.3 

4.2 

4.2 

-  Imagemap  (click  on  correct  part  of  image) 

4.9 

4.5 

4.0 

4.8 

-  Short  answer 

5.0 

3.5 

4.4 

4.5 

Questions  can  have  multiple  correct  answers 

4.7 

3.6 

4.4 

4.3 

Can  use  a  mixture  of  question  types  on  a  single  quiz 

5.4 

4.7 

5.0 

5.0 

One  question  at-a-time  testing  capability 

5.1 

4.5 

4.6 

5.3 

Question  fde  upload  capability 

5.3 

4.1 

4.8 

4.5 

Customized  feedback 

5.2 

4.6 

4.7 

4.8 

Redirect  path  dependent  on  question  answers 

4.5 

3.7 

4.2 

4.5 

Timed  quizzes 

4.6 

3.4 

3.8 

3.7 

Delivered  on-line  on  a  predetermined  time  and  day 

4.7 

3.9 

4.5 

4.4 

Supports  graphics  fdes  adjacent  to  quiz  question 

4.9 

4.5 

4.9 

4.2 

Supports  both  tutorial  and  real  exam  scenarios 

5.0 

4.7 

4.3 

4.5 

Random  assignment  of  questions  to  exams 

4.7 

4.3 

3.9 

4.6 

Allows  weighting  of  questions  so  students  get  equal  quizzes 

5.0 

4.7 

3.8 

4.6 

Other: 

Can  be  used  with  the  following  server  /Operating  Systems 

Unix 

3.5 

2.6 

2.8 

3.2 

NT 

5.1 

3.8 

4.5 

3.7 

Macintosh 

2.6 

2.8 

2.0 

3.3 

Solaris 

2.6 

2.2 

2.0 

3.3 

Linux 

2.7 

3.0 

2.5 

3.3 

Other: 

